{"@attributes":{"version":"2.0"},"channel":{"title":"Chaoming's Blog","link":"https:\/\/chaoming.li\/","description":"Recent content on Chaoming's Blog","generator":"Hugo -- gohugo.io","language":"en-us","lastBuildDate":"Mon, 05 Jan 2026 00:00:00 +0000","item":[{"title":"AI in SaaS in 2026: A Founder's Perspective","link":"https:\/\/chaoming.li\/blog\/ai-in-saas-in-2026-a-founders-perspective\/","pubDate":"Mon, 05 Jan 2026 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/ai-in-saas-in-2026-a-founders-perspective\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/ai-in-saas-in-2026-a-founders-perspective\/saas-ai-agent.jpg\" alt=\"Featured image of post AI in SaaS in 2026: A Founder's Perspective\" \/><p>2025 was a breakout year for AI in my workflow. I started using AI for coding assistance in 2024, but by 2025 it was no longer just generating snippets. It began shaping architecture decisions, accelerating execution, and helping move real projects forward.<\/p>\n<p>That shift naturally leads to a broader question: <strong>what does AI mean for SaaS products in 2026?<\/strong><\/p>\n<p>Below are my perspectives as a founder building and using SaaS every day.<\/p>\n<h2 id=\"from-answers-to-actions\">From answers to actions\n<\/h2><p>AI in SaaS started the same way AI in coding did. You ask a question and get an answer. Early copilots focused on explanations, suggestions, and surface-level assistance.<\/p>\n<p>Coding tools did not stop there. They evolved into systems that draft code, wire components together, generate tests, open pull requests, and iterate alongside developers. SaaS is following the same trajectory.<\/p>\n<p>In 2026, the most valuable AI copilots will not simply explain how to use a product. They will take action on behalf of the user, guided by intent and constrained by clear controls.<\/p>\n<p>Consider a digital marketer who wants to compare mobile versus desktop performance for a Boxing Day campaign. Today, this usually means navigating multiple reports, applying filters, selecting segments, and exporting charts. Instead, the user should be able to say:<\/p>\n<blockquote>\n<p>\u201cCompare mobile vs desktop performance for Boxing Day. Explain what changed and why.\u201d<\/p>\n<\/blockquote>\n<p>The agent builds the report, applies the correct filters, pulls the relevant data, and summarizes the outcome.<\/p>\n<p>This direction is already visible in the market. Gartner predicts that 40 percent of enterprise applications will include task-specific AI agents by 2026, up from single-digit adoption today. Leading developer tools show that moving from suggestion to execution dramatically increases daily usage and stickiness. The shift is not about novelty. It is about reducing time-to-outcome.<\/p>\n<h2 id=\"trust-and-approval-become-the-real-bottleneck\">Trust and approval become the real bottleneck\n<\/h2><p>As AI agents gain the ability to act, trust becomes the central product challenge. Users will want to understand what the agent is doing, why it is doing it, what data it used, and which actions require approval.<\/p>\n<p>From a product standpoint, fully opaque agents will struggle to gain adoption in serious B2B software. This is not only a UX concern. Governance and regulation are already reinforcing this expectation.<\/p>\n<p>The NIST AI Risk Management Framework emphasizes transparency, explainability, and human oversight as core principles for trustworthy AI. Regulatory efforts such as the EU AI Act further highlight the importance of human oversight for systems that can materially affect outcomes.<\/p>\n<p>In practice, the winning pattern will look less like a generic chat window and more like an embedded copilot. One that operates inside the product UI, shows intermediate steps, previews changes, and asks for confirmation when the stakes are high. Trust will be earned through visibility.<\/p>\n<h2 id=\"personalization-is-how-agents-become-genuinely-useful\">Personalization is how agents become genuinely useful\n<\/h2><p>An agent that treats every user the same will feel shallow very quickly. In B2B SaaS especially, usefulness depends on context.<\/p>\n<p>That context includes company goals and KPIs, the user\u2019s role and seniority, permissions and risk tolerance, and preferred level of detail. A CFO and a marketing analyst can look at the same dataset and expect entirely different assistance. One wants summarized impact. The other wants drill-downs and anomalies.<\/p>\n<p>Agents only begin to feel intelligent when they adapt to the person, not just the data. This is less about model capability and more about thoughtful product design.<\/p>\n<h2 id=\"why-ai-agents-in-saas-become-non-optional\">Why AI agents in SaaS become non-optional\n<\/h2><p>The strongest case for agents is not hype. It is productivity.<\/p>\n<p>When AI is embedded directly into workflows, the gains are measurable. GitHub has reported that developers using Copilot completed certain tasks roughly 55 percent faster in controlled experiments. In a real customer support environment, a well-cited field study found approximately 14 percent higher productivity, with even larger gains among less experienced workers. Research in consulting and knowledge work shows similar patterns, while also highlighting that poor integration can reduce quality.<\/p>\n<p>Translated into SaaS terms, this has several practical implications.<\/p>\n<ul>\n<li><strong>A lower learning curve leads to higher adoption.<\/strong> Instead of reading documentation or booking training sessions, users can ask an agent to perform tasks while observing how they are done. Value is delivered immediately, and learning happens implicitly.<\/li>\n<li><strong>Support costs also shift.<\/strong> When agents handle routine tasks such as generating reports, locating invoices, or configuring dashboards, support teams can focus on complex and strategic issues that drive retention and expansion.<\/li>\n<li><strong>Ultimately, speed becomes the differentiator.<\/strong> SaaS exists to help users get jobs done. Products that minimize friction and compress time-to-outcome will consistently outperform those that rely on manual navigation and configuration. By 2026, software that requires extensive clicking for workflows will feel dated.<\/li>\n<\/ul>\n<h2 id=\"the-tools-layer-matters-more-than-the-model\">The tools layer matters more than the model\n<\/h2><p>The effectiveness of AI agents depends less on model size and more on the tools and context they can safely access. A powerful model without the right tools behaves like a smart assistant without hands or memory. It can explain, but it cannot reliably help users get work done.<\/p>\n<p>Three elements stand out as foundational.<\/p>\n<ul>\n<li><strong>UI-driven actions<\/strong>\nThe first is UI-driven action. The agent operates the product interface the way an experienced user would, opening reports, applying filters, configuring settings, and building dashboards in full view of the user.\nThis approach has two benefits. It makes actions transparent, and it aligns the agent\u2019s behavior with how the product is actually meant to be used. For sensitive operations, the agent can prepare everything while leaving the final confirmation to the user.<\/li>\n<li><strong>API-driven execution<\/strong>\nThe second is API-driven execution. For scale, performance, and reliability, agents need structured APIs and tool interfaces rather than brittle UI automation alone.\nThis layer is where permissions, validation rules, and audit logging live. Well-designed APIs allow agents to act confidently while staying within clearly defined boundaries.<\/li>\n<li><strong>Product knowledge as first-class input<\/strong>\nFor an AI agent to be genuinely useful, it must behave like an expert user of the SaaS product it supports. That requires access to the product\u2019s knowledge base, not just raw data or APIs.\nThis includes documentation, feature explanations, best practices, limitations, and common workflows. Without this context, an agent may technically be able to take actions, but it will struggle to choose the right ones or sequence them correctly to solve real-world problems.\nIn practice, this means agents should be guided by the same institutional knowledge that experienced human users rely on. When a user asks how to solve a problem, the agent should not only execute steps but also understand which features are appropriate, which trade-offs exist, and which approach aligns with the product\u2019s intended usage.\nTreating the knowledge base as a first-class input allows agents to move beyond mechanical execution and toward informed decision-making.<\/li>\n<li><strong>Guardrails by design<\/strong>\nFinally, agents must be paired with strong guardrails. If agents are expected to act, they need role-based permissions, previews before changes are applied, detailed action logs, reversible operations, and escalation paths when confidence is low.\nThese controls do not slow agents down. They make adoption possible. Without them, increased capability quickly turns into increased risk.<\/li>\n<\/ul>\n<h2 id=\"every-saas-becomes-an-ai-business\">Every SaaS becomes an AI business\n<\/h2><p>AI feels similar to earlier waves of digital transformation. At first, it is a differentiator. Then it becomes table stakes.<\/p>\n<p>In my view, AI will follow the same path. By 2026, it will no longer be framed as a feature or an add-on. It will be part of the core value proposition of every serious SaaS product.<\/p>\n<p>User expectations are shifting away from features and toward outcomes. Agents are the most direct way to bridge that gap.<\/p>\n<p><strong>How are you thinking about introducing AI to your SaaS users?<\/strong><\/p>"},{"title":"Introducing fireact.dev: An Open Source SaaS Framework Built with AI","link":"https:\/\/chaoming.li\/blog\/introducing-fireact.dev-an-open-source-saas-framework-built-with-ai\/","pubDate":"Fri, 22 Nov 2024 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/introducing-fireact.dev-an-open-source-saas-framework-built-with-ai\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/introducing-fireact.dev-an-open-source-saas-framework-built-with-ai\/fireact.png\" alt=\"Featured image of post Introducing fireact.dev: An Open Source SaaS Framework Built with AI\" \/><p>I&rsquo;m excited to introduce <a class=\"link\" href=\"https:\/\/www.fireact.dev\" target=\"_blank\" rel=\"noopener\"\n>fireact.dev<\/a>, an open-source SaaS framework that brings together the power of React, Firebase, and Stripe to revolutionize how developers build subscription-based web applications. As someone deeply passionate about creating technology that empowers others, I built fireact.dev to address the common challenges developers face when launching SaaS products.<\/p>\n<h2 id=\"what-is-fireactdev\">What is fireact.dev?\n<\/h2><p>fireact.dev is a comprehensive framework that combines three powerful technologies:<\/p>\n<ul>\n<li><strong>React<\/strong>: For building dynamic and responsive user interfaces<\/li>\n<li><strong>Firebase<\/strong>: For secure authentication and scalable backend services<\/li>\n<li><strong>Stripe<\/strong>: For handling subscriptions and payments<\/li>\n<\/ul>\n<p>The framework provides developers with a solid foundation to build subscription-based SaaS applications without starting from scratch. What makes fireact.dev unique is that both the framework itself and its documentation website were entirely generated using AI technology, showcasing the potential of AI in modern software development.<\/p>\n<h2 id=\"key-features\">Key Features\n<\/h2><ol>\n<li><strong>Authentication System<\/strong>: Pre-built user authentication powered by Firebase<\/li>\n<li><strong>Subscription Management<\/strong>: Integrated Stripe subscription handling<\/li>\n<li><strong>User Management<\/strong>: Complete user administration interface<\/li>\n<li><strong>Role-based Access Control<\/strong>: Built-in permission system<\/li>\n<li><strong>Responsive Design<\/strong>: Mobile-friendly interface out of the box<\/li>\n<li><strong>Modern Tech Stack<\/strong>: Latest React features and best practices<\/li>\n<\/ol>\n<h2 id=\"who-should-use-fireactdev\">Who Should Use fireact.dev?\n<\/h2><p>fireact.dev is perfect for:<\/p>\n<ul>\n<li><strong>Developers<\/strong> looking to launch SaaS products quickly<\/li>\n<li><strong>Startups<\/strong> wanting to validate their ideas without heavy investment<\/li>\n<li><strong>Teams<\/strong> needing a solid foundation for subscription-based services<\/li>\n<li><strong>Solo entrepreneurs<\/strong> building their next big thing<\/li>\n<\/ul>\n<h2 id=\"the-power-of-ai-in-development\">The Power of AI in Development\n<\/h2><p>What makes fireact.dev particularly interesting is that it&rsquo;s a testament to the capabilities of AI in software development. The entire project, including:<\/p>\n<ul>\n<li>Core framework code<\/li>\n<li>Component architecture<\/li>\n<li>Documentation<\/li>\n<li>Website content<\/li>\n<\/ul>\n<p>was generated through AI assistance. This not only showcases the potential of AI in software development but also ensures that the framework incorporates modern best practices and patterns.<\/p>\n<h2 id=\"my-saas-journey\">My SaaS Journey\n<\/h2><p>Building fireact.dev stems from my passion for creating tools that help others succeed. Throughout my career, I&rsquo;ve been involved in various SaaS projects, including:<\/p>\n<ul>\n<li>Insightech: A platform for understanding user behavior and its impact on conversions<\/li>\n<li>VisitorAPI: A fast and reliable API for visitor detection<\/li>\n<li>Custom web analytics solutions<\/li>\n<\/ul>\n<p>These experiences helped me understand the common challenges in SaaS development and inspired me to create a framework that makes it easier for others to bring their ideas to life.<\/p>\n<h2 id=\"getting-started\">Getting Started\n<\/h2><p>fireact.dev is open source and freely available for anyone to use. Whether you&rsquo;re building your first SaaS product or your tenth, fireact.dev provides the foundation you need to focus on what matters most \u2013 creating value for your users.<\/p>\n<p>Visit <a class=\"link\" href=\"https:\/\/www.fireact.dev\" target=\"_blank\" rel=\"noopener\"\n>fireact.dev<\/a> to explore the documentation, try out the demo, and start building your next SaaS project with a framework that combines the best of React, Firebase, and Stripe, all brought together through the power of AI.<\/p>\n<p>Join us in revolutionizing how SaaS applications are built, and let&rsquo;s create something amazing together.<\/p>"},{"title":"How to trigger marketing tags based on visitor location in GTM","link":"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/","pubDate":"Thu, 07 Jul 2022 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/syj23egvntavpv64up1t.webp\" alt=\"Featured image of post How to trigger marketing tags based on visitor location in GTM\" \/><p>Location variables are not built-in variables in Google Tag Manager (GTM). This article will show you how you can trigger marketing tags based on visitor location by using VisitorAPI in GTM.<\/p>\n<p>First, import the VisitorAPI tag template to your GTM container by clicking <strong>Templates<\/strong> \u2192 <strong>Search Gallery<\/strong>, then search for <a class=\"link\" href=\"https:\/\/tagmanager.google.com\/gallery\/#\/owners\/visitorapi\/templates\/visitor-api-google-tag-manager\" target=\"_blank\" rel=\"noopener\"\n>\u201cvisitorapi\u201d<\/a> to import the tag template to your GTM container.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/gtm-visitor-api.png\"\nwidth=\"775\"\nheight=\"180\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/gtm-visitor-api_hu_3bcbb601080b0bdb.png 480w, https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/gtm-visitor-api_hu_35ecb8d912cd96bd.png 1024w\"\nloading=\"lazy\"\nalt=\"Import VisitorAPI template in Google Tag Manager\"\nclass=\"gallery-image\"\ndata-flex-grow=\"430\"\ndata-flex-basis=\"1033px\"\n><\/p>\n<p>Once the tag template is imported, create a new tag with the <a class=\"link\" href=\"https:\/\/www.visitorapi.com\/\" target=\"_blank\" rel=\"noopener\"\n>VisitorAPI<\/a> tag template, and put in your VisitorAPI project ID. To get a VisitorAPI project ID, sign up for an account and create a free or paid project in VisitorAPI. Don\u2019t forget to add your website domains to the authorised domain list in your VisitorAPI project to make the API works for your website.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-visitor-api-tag.png\"\nwidth=\"957\"\nheight=\"594\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-visitor-api-tag_hu_30e4ac9d978e21b.png 480w, https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-visitor-api-tag_hu_f41bb9a1f62050df.png 1024w\"\nloading=\"lazy\"\nalt=\"Create VisitorAPI tag in Google Tag Manager\"\nclass=\"gallery-image\"\ndata-flex-grow=\"161\"\ndata-flex-basis=\"386px\"\n><\/p>\n<p>The VisitorAPI tag will call the API to retrieve the visitor\u2019s location data including country, state and city, and trigger a new event \u201cvisitor-api-success\u201d which you can use to trigger the tags that are targeting certain locations.<\/p>\n<p>To use the location variables, you will need to create them in your GTM container. VisitorAPI returns the country, state and city of the visitors. Here is how to create those three variables: click <strong>Variables<\/strong> \u2192 <strong>New<\/strong>, then select <strong>Data Layer Variable<\/strong>. Name your variables as \u201cCountry Code\u201d and input \u201cvisitorApiCountryCode\u201d to the <strong>Data Layer Variable Name<\/strong> field.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-visitor-api-variables.png\"\nwidth=\"1742\"\nheight=\"557\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-visitor-api-variables_hu_ac52e67a446f659a.png 480w, https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-visitor-api-variables_hu_96ecb4b9e99996f1.png 1024w\"\nloading=\"lazy\"\nalt=\"Create VisitorAPI variables in Google Tag Manager\"\nclass=\"gallery-image\"\ndata-flex-grow=\"312\"\ndata-flex-basis=\"750px\"\n><\/p>\n<p>Here is a list of all the variables VisitorAPI returns<\/p>\n<table>\n<thead>\n<tr>\n<th>GTM Variable Name<\/th>\n<th>Data Layer Variable Name<\/th>\n<th>Description<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>IP Address<\/td>\n<td>visitorApiIpAddress<\/td>\n<td>The IP address of the visitor.<\/td>\n<\/tr>\n<tr>\n<td>Country Code<\/td>\n<td>visitorApiCountryCode<\/td>\n<td>The country from which the visitor is located as an ISO 3166-1 alpha-2 country code.<\/td>\n<\/tr>\n<tr>\n<td>Country Name<\/td>\n<td>visitorApiCountryName<\/td>\n<td>The full name of the country in which the visitor is located in.<\/td>\n<\/tr>\n<tr>\n<td>Currencies<\/td>\n<td>visitorApiCurrencies<\/td>\n<td>An array of the official currencies of the country in which the visitor is located.<\/td>\n<\/tr>\n<tr>\n<td>Languages<\/td>\n<td>visitorApiLanguages<\/td>\n<td>An array of the official languages of the country in which the visitor is located in.<\/td>\n<\/tr>\n<tr>\n<td>Region<\/td>\n<td>visitorApiRegion<\/td>\n<td>Name of the region, state or province in which the visitor is located. The complete list of valid region values is found in the ISO-3166-2 standard.<\/td>\n<\/tr>\n<tr>\n<td>City<\/td>\n<td>visitorApiCity<\/td>\n<td>Name of the city in which the visitor is located in.<\/td>\n<\/tr>\n<tr>\n<td>City LatLong<\/td>\n<td>visitorApiCityLatLong<\/td>\n<td>Latitude and longitude of the city in which the visitor is located in.<\/td>\n<\/tr>\n<tr>\n<td>Browser<\/td>\n<td>visitorApiBrowser<\/td>\n<td>The browser name which the visitor uses.<\/td>\n<\/tr>\n<tr>\n<td>Browser Version<\/td>\n<td>visitorApiBrowserVersion<\/td>\n<td>The browser version which the visitor uses.<\/td>\n<\/tr>\n<tr>\n<td>Device Brand<\/td>\n<td>visitorApiDeviceBrand<\/td>\n<td>The brand of the device which the visitor uses. Only applicable to mobile devices.<\/td>\n<\/tr>\n<tr>\n<td>Device Model<\/td>\n<td>visitorApiDeviceModel<\/td>\n<td>The model of the device which the visitor uses. Only applicable to mobile devices.<\/td>\n<\/tr>\n<tr>\n<td>Device Family<\/td>\n<td>visitorApiDeviceFamily<\/td>\n<td>The family of the device which the visitor uses. Only applicable to mobile devices.<\/td>\n<\/tr>\n<tr>\n<td>OS<\/td>\n<td>visitorApiOs<\/td>\n<td>The operating system name of the device which the visitor uses.<\/td>\n<\/tr>\n<tr>\n<td>OS Version<\/td>\n<td>visitorApiOsVersion<\/td>\n<td>The operating system version of the device which the visitor uses.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>You can use the variables in trigger conditions in your GTM container to fire the tags targeting visitors based on locations. For example, if you want to target visitors in the US, you can create a trigger with the condition \u201cCountry Code\u201d equals \u201cus\u201d.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-tag-for-us-country.png\"\nwidth=\"1740\"\nheight=\"559\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-tag-for-us-country_hu_389e15bb631bdfa7.png 480w, https:\/\/chaoming.li\/blog\/how-to-trigger-marketing-tags-based-on-visitor-location-in-gtm\/create-tag-for-us-country_hu_e5cd8cd3de530307.png 1024w\"\nloading=\"lazy\"\nalt=\"Create a marketing tag for the US visitors in Google Tag Manager\"\nclass=\"gallery-image\"\ndata-flex-grow=\"311\"\ndata-flex-basis=\"747px\"\n><\/p>\n<p>Not only you can trigger tags based on the location variables, but you can also use the variables in your tags as parameters <code>{{Country Code}}<\/code> to pass the data to your tags.<\/p>\n<p>The location variables make it possible to personalize user experience based on visitor location by using GTM. Please feel free to leave a comment to share how you will use the location variables in GTM.<\/p>"},{"title":"Build a Static Blog with Hugo","link":"https:\/\/chaoming.li\/blog\/build-a-static-blog-with-hugo\/","pubDate":"Sat, 25 Jun 2022 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/build-a-static-blog-with-hugo\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/build-a-static-blog-with-hugo\/hugo.png\" alt=\"Featured image of post Build a Static Blog with Hugo\" \/><h2 id=\"why-static-website-builder\">Why Static Website Builder\n<\/h2><p>I had a blog that was using WordPress and it was hosted on an AWS EC2 + CloudFront CDN. It was fast and not too hard to use, except that sometimes it might break down because of a WordPress update or issues with the MySQL database. Since I built the <a class=\"link\" href=\"http:\/\/fireact.dev\" target=\"_blank\" rel=\"noopener\"\n>fireact.dev<\/a> website with Jekyll, I have been thinking to move the old blog site to a static solution because:<\/p>\n<ul>\n<li>No need for a database, which means less maintenance headache<\/li>\n<li>Free hosting options like Github and Firebase<\/li>\n<li>Fast, super fast! That is good for UX and SEO<\/li>\n<\/ul>\n<h2 id=\"the-solution\">The Solution\n<\/h2><p>I did some research and decided not to use Jekyll because I found <a class=\"link\" href=\"https:\/\/gohugo.io\/\" target=\"_blank\" rel=\"noopener\"\n>Hugo<\/a> is gaining popularity and it has a lot of free themes too. I picked a theme called <a class=\"link\" href=\"https:\/\/themes.gohugo.io\/themes\/hugo-theme-stack\/\" target=\"_blank\" rel=\"noopener\"\n>Stack<\/a>, which looks great in my opinion and is well maintained by its creator.<\/p>\n<p>Creating a Hugo site is quite simple following its step-by-step instruction document. The themes are installed as <code>git submodules<\/code> which makes them relatively simple to install.<\/p>\n<p>Hugo also has a deployment config for Github pages so you can deploy your website by pushing commits to a branch.<\/p>\n<h2 id=\"writing-posts-with-markdown\">Writing Posts with Markdown\n<\/h2><p>The most common way to write content for static websites is to create markdown files. Markdown is widely used in writing documentation for tech projects. You don\u2019t need to worry about HTML tags, just write with some formatting syntax like \u201c##\u201d means <h2> tag in HTML.<\/p>\n<p>If you don\u2019t want to write with markdown syntax, <a class=\"link\" href=\"https:\/\/www.notion.so\/\" target=\"_blank\" rel=\"noopener\"\n>Notion<\/a> is a great tool. It has a very user-friendly interface for writing content. Once you finish an article, select the whole article and copy it, and you can paste content to a markdown file.<\/p>\n<h2 id=\"my-thoughts\">My Thoughts\n<\/h2><p>There are many open-source projects are using static website builders for building websites for the projects and are hosting the websites on Github.<\/p>\n<p>I think it\u2019s totally possible to use the same solution for early-stage startups to build their websites and host them on Firebase. For SaaS startup, my thinking is that a static website for the content, and <a class=\"link\" href=\"http:\/\/fireact.dev\" target=\"_blank\" rel=\"noopener\"\n>fireact.dev<\/a> for the web application. All these can be hosted on Firebase with minimum costs.<\/p>\n<p>Although Notion is a great tool for writing and converting content to markdown format, I think there is a need for some kind of CMS application that can do a similar job and automate the process to update and deploy static websites. Or maybe just automate the deployment process of Notion articles to static websites, because Notion is already doing a really good job as a writing tool. So that people who don\u2019t know much about Git can also easily update their static websites.<\/p>"},{"title":"Automate Deployments to Multiple App Engine Environments with Cloud Build and GitHub","link":"https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/","pubDate":"Tue, 16 Jul 2019 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/1_UzYiagjfQwOZTlIwmCqKOQ.png\" alt=\"Featured image of post Automate Deployments to Multiple App Engine Environments with Cloud Build and GitHub\" \/><p>If you want to securely automate deployments of your App Engine application to multiple environments such as staging and production with a simple Git commit or a pull request, this article will outline the steps for you.<\/p>\n<h2 id=\"why-you-should-do-this\">Why You Should Do This\n<\/h2><p>Good reasons to automate the deployment process are:<\/p>\n<ul>\n<li>Reduce the impact of human error on a deployment<\/li>\n<li>Improve security by not including any credentials or sensitive configuration settings in the source code<\/li>\n<li>Restrict permission to deploy to certain environments (e.g. production) with a process<\/li>\n<li>Streamline the deployment process and gain the benefits of automation (e.g. testing)<\/li>\n<\/ul>\n<h2 id=\"assumptions\">Assumptions\n<\/h2><p>I am not going to discuss how to set up Git repositories and GitHub accounts. There are plenty of tutorials online for those topics.<\/p>\n<p>In this article, I\u2019ll only assume that you have an App Engine application. I have used this process for Golang and PHP, but it will work for Python and other languages.<\/p>\n<h2 id=\"step-1-enable-branch-restrictions\">Step 1: Enable Branch Restrictions\n<\/h2><p>In your GitHub repository, you need to have some branches, at least a staging branch and a master branch. The workflow is that a developer can commit to the staging branch. It will automatically deploy the application to a staging App Engine environment.<\/p>\n<p>After the application is tested in the staging environment:<\/p>\n<ul>\n<li>The developer will create a pull request<\/li>\n<li>The administrator will merge it with the master branch<\/li>\n<li>The automation will deploy the application to the production environment<\/li>\n<\/ul>\n<p>This workflow is a bit oversimplified. But the point is: Only someone with permission can deploy the application to the production by committing to the master branch. This can be achieved with <a class=\"link\" href=\"https:\/\/help.github.com\/en\/articles\/enabling-branch-restrictions\" target=\"_blank\" rel=\"noopener\"\n>branch restrictions<\/a>.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/1_UzYiagjfQwOZTlIwmCqKOQ.png\"\nwidth=\"837\"\nheight=\"461\"\nsrcset=\"https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/1_UzYiagjfQwOZTlIwmCqKOQ_hu_f75668e481ffc3d.png 480w, https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/1_UzYiagjfQwOZTlIwmCqKOQ_hu_be62a930a5918ee4.png 1024w\"\nloading=\"lazy\"\nalt=\"Deployment Workflow Diagram\"\nclass=\"gallery-image\"\ndata-flex-grow=\"181\"\ndata-flex-basis=\"435px\"\n><\/p>\n<h2 id=\"step-2-move-all-environment-related-configurations-to-appyaml\">Step 2: Move All Environment Related Configurations to app.yaml\n<\/h2><p>It\u2019s very common for an application to have environment variables for configuration and secrets such as database credentials, domains, GCP key files, etc. Some of these details are security-critical so they should never be included in the source code. If you have embedded them into part of the source code, it\u2019s time to remove them and put them into the app.yaml file that is kept outside of the repository and only is used during deployment.<\/p>\n<p>In your app.yaml file, you can have these variables as environment variables in the following syntax:<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\">1\n<\/span><span class=\"lnt\">2\n<\/span><span class=\"lnt\">3\n<\/span><span class=\"lnt\">4\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-yaml\" data-lang=\"yaml\"><span class=\"line\"><span class=\"cl\"><span class=\"nt\">env_variables<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">DOMAIN<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;staging.mydomain.com&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">DEBUG_MODE<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"kc\">true<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">MYSQL_DSN<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;blah blah&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>The App Engine platform will take these values from app.yaml and make them available to your application, which can read the environment variables to ensure your application has the correct settings.<\/p>\n<p>Depending on which language your application is written in, there are different ways to read the environment variables. Here is the link to Python and other languages (click on the language links at the top of the content to switch to your preferred language): <a class=\"link\" href=\"https:\/\/cloud.google.com\/appengine\/docs\/standard\/python\/config\/appref\" target=\"_blank\" rel=\"noopener\"\n>https:\/\/cloud.google.com\/appengine\/docs\/standard\/python\/config\/appref<\/a><\/p>\n<p>In PHP, use:<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\">1\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-php\" data-lang=\"php\"><span class=\"line\"><span class=\"cl\"><span class=\"nx\">getenv<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;VAR NAME&#39;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>I don\u2019t think $_ENV works.<\/p>\n<p>If you need app.yaml for your local environment, just make sure the file is in .gitignore so it won\u2019t be included in the repository.<\/p>\n<p>Once you have put the environment variables into the staging app.yaml file, upload it to a Google Cloud Storage bucket and make sure it\u2019s not available to the public (by default it isn\u2019t). Do the same for the production app.yaml but use a different bucket.<\/p>\n<h2 id=\"step-3-create-your-cloud-build-for-staging\">Step 3: Create Your Cloud Build for Staging\n<\/h2><p>In GCP, Cloud Build can trigger a deployment from Git commits to a certain branch. The web UI is very simple to follow to create the trigger. The trick is to have a variable called _BUCKET which is the bucket name that contains your app.yaml<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/1_GBG2F-EdARSeVptNoYNo2Q.png\"\nwidth=\"501\"\nheight=\"597\"\nsrcset=\"https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/1_GBG2F-EdARSeVptNoYNo2Q_hu_63643ceed242d99e.png 480w, https:\/\/chaoming.li\/blog\/automate-deployments-to-multiple-app-engine-environments-with-cloud-build-and-github\/1_GBG2F-EdARSeVptNoYNo2Q_hu_a1fb954e8ecfc5b3.png 1024w\"\nloading=\"lazy\"\nalt=\"Cloud Build Screenshot\"\nclass=\"gallery-image\"\ndata-flex-grow=\"83\"\ndata-flex-basis=\"201px\"\n><\/p>\n<p>And, you need a cloudbuild.yaml file in your Git repository to control the deployment process. Below is a simple cloudbuild.yaml file that does two things:<\/p>\n<ul>\n<li>Copy the app.yaml file from the bucket<\/li>\n<li>Deploy the application<\/li>\n<\/ul>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\">1\n<\/span><span class=\"lnt\">2\n<\/span><span class=\"lnt\">3\n<\/span><span class=\"lnt\">4\n<\/span><span class=\"lnt\">5\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-yaml\" data-lang=\"yaml\"><span class=\"line\"><span class=\"cl\"><span class=\"nt\">steps<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\">- <span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l\">gcr.io\/cloud-builders\/gsutil<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">args<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;cp&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gs:\/\/$_BUCKET\/app.yaml&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;html\/app.yaml&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\">- <span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gcr.io\/cloud-builders\/gcloud&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">args<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;app&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;deploy&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;html\/&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>You can see that I use $_BUCKET in the cloudbuild.yaml file as the bucket name of the app.yaml file. This means when I configure the production Cloud Build trigger, the same cloudbuild.yaml will work without any modification.<\/p>\n<p>Now, commit to your staging branch, and you will see Cloud Build is triggered and starts the deployment process automatically.<\/p>\n<h2 id=\"step-4-grant-permissions-to-cloud-build\">Step 4: Grant Permissions to Cloud Build\n<\/h2><p>The deployment is likely to fail because Cloud Build doesn\u2019t have the permissions to access Cloud Storage, App Engine, etc.<\/p>\n<p>Go to IAM web UI in your GCP project, and change the permission settings of the Cloud Build service account to grant it App Engine Admin and other relevant permissions for completing the deployment. Now, retry the deployment in Cloud Build, you should see the successfully deployed message in the logs.<\/p>\n<h2 id=\"step-5-repeat-step-3-and-step-4-for-production\">Step 5: Repeat Step 3 and Step 4 for Production\n<\/h2><p>Just repeat the above two steps in the production project. Make sure the production app.yaml is in the Cloud Storage bucket for production, and the Cloud Build trigger _BUCKET variable is pointing to the production bucket as well.<\/p>\n<p>Because only the system administrator is able to commit to production, as a developer, you should create a Pull Request in the GitHub repository to merge your changes from the staging branch into the master branch. Once the Pull Request is approved and merged, it will trigger the Cloud Build deployment process in the production project.<\/p>\n<p>Conclusion:<\/p>\n<p>This approach not only streamlines the deployment process, but it also makes it more secure and reliable.<\/p>\n<p>If you have any feedback or questions, please feel free to comment below.<\/p>"},{"title":"How to Install Custom SSL Certificate on Google Cloud Platform","link":"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/","pubDate":"Fri, 24 May 2019 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_NG0OmIlxq8M686LHMQHJvg.jpeg\" alt=\"Featured image of post How to Install Custom SSL Certificate on Google Cloud Platform\" \/><p>I recently installed custom SSL certificates in Google Cloud Platform (GCP) for both App Engine and load balancer (used by CDN and Computer Engine). Here is the step-by-step guide on how it\u2019s done. I hope this will help you to go through the process.<\/p>\n<h2 id=\"generating-a-certificate-signing-request\">Generating a certificate signing request\n<\/h2><p>To get a certificate, firstly, you need to generate a certificate signing request to your SSL certificate issuer so that they can provide you with a valid certificate. Here is the command using OpenSSL to do it.<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\">1\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr\n<\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>Input the relevant information and put your domain name in Common Name, it will generate two files: server.key which is your private key, and server.csr which is the certificate signing request.<\/p>\n<h2 id=\"get-your-ssl-certificate\">Get your SSL certificate\n<\/h2><p>Provide your certificate issuer with your CSR file, they will generate your SSL certificate. Usually, this process only takes a few minutes. However, the process varies depending on which issuer you are going to use.<\/p>\n<p>The format of the certificate can also be different depending on the issuer. I got the X.509 certificate in a crt file in this case.<\/p>\n<h2 id=\"convert-your-private-key-to-pem-format\">Convert your private key to PEM format\n<\/h2><p>If you only need to put your certificate in the GCP load balancer, you can skip this step as it doesn\u2019t need a PEM format certificate. This is for App Engine. I don\u2019t get why Google can\u2019t just let me manage all my certificates in one place and in one format. This is actually a bit annoying.<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\">1\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-fallback\" data-lang=\"fallback\"><span class=\"line\"><span class=\"cl\">openssl rsa -in server.key -out server.key.pem\n<\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><h2 id=\"upload-certificate-and-pem-key-to-app-engine\">Upload certificate and PEM key to App Engine\n<\/h2><p>In the App Engine settings interface, click on the SSL certificates tab and you can upload a certificate.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_tjA2bdDStYqfhoMi_ODPog.png\"\nwidth=\"526\"\nheight=\"97\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_tjA2bdDStYqfhoMi_ODPog_hu_4d2b3bb9183495a0.png 480w, https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_tjA2bdDStYqfhoMi_ODPog_hu_5e3107d86bffc414.png 1024w\"\nloading=\"lazy\"\nalt=\"App Engine Settings Screenshot\"\nclass=\"gallery-image\"\ndata-flex-grow=\"542\"\ndata-flex-basis=\"1301px\"\n><\/p>\n<p>That will bring up this interface where you can copy and paste your certificate and private key.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_rQaHJYhngjpX7luLv8FCYw.png\"\nwidth=\"511\"\nheight=\"482\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_rQaHJYhngjpX7luLv8FCYw_hu_6c69e95e5767b0ad.png 480w, https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_rQaHJYhngjpX7luLv8FCYw_hu_2ecd1efce6536937.png 1024w\"\nloading=\"lazy\"\nalt=\"Add SSL Certificate to App Engine Screenshot\"\nclass=\"gallery-image\"\ndata-flex-grow=\"106\"\ndata-flex-basis=\"254px\"\n><\/p>\n<p>Give your certificate a name. I usually put the year it is issued in it so that when I need to renew it next year, I will have a good idea of which certificate is for which year.<\/p>\n<p>Copy and paste your X.509 certificate file content and the PEM format private key content in the two boxes. Click \u201cUpload\u201d to save the certificate and key. It will ask you to enable SSL for the custom domains you have in App Engine. Just pick the relevant domains of the certificate.<\/p>\n<p>This change will take about 10 minutes or more to become effective. You can verify that by checking your domain\u2019s SSL certificate with any browser. It\u2019s better to check it with all major browsers to make sure they like your certificate.<\/p>\n<h2 id=\"upload-certificate-to-the-gcp-load-balancer\">Upload certificate to the GCP load balancer\n<\/h2><p>To upload a certificate to the GCP load balancer, you need to go to Network services then edit your load balancer. Then click on Frontend configuration and you will be able to edit the port 443 configuration settings where you can manage your SSL certificates.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_rmv2F1zrdyZy-r2xALDTgQ.png\"\nwidth=\"647\"\nheight=\"483\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_rmv2F1zrdyZy-r2xALDTgQ_hu_a58073311f5565fb.png 480w, https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_rmv2F1zrdyZy-r2xALDTgQ_hu_e2fce34031d85c05.png 1024w\"\nloading=\"lazy\"\nalt=\"Load Balancer Screenshot\"\nclass=\"gallery-image\"\ndata-flex-grow=\"133\"\ndata-flex-basis=\"321px\"\n><\/p>\n<p>After clicking on \u201cCreate a new certificate\u201d in the Certificate dropdown list, you will see the interface to copy and paste in your certificate, certificate chain (root certificate), and the private key.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_IJ5ctnCd-_EEuAOzga3hjw.png\"\nwidth=\"479\"\nheight=\"610\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_IJ5ctnCd-_EEuAOzga3hjw_hu_3d7ebea9c1a397ba.png 480w, https:\/\/chaoming.li\/blog\/how-to-install-custom-ssl-certificate-on-google-cloud-platform\/1_IJ5ctnCd-_EEuAOzga3hjw_hu_d5846bb6e0911b8.png 1024w\"\nloading=\"lazy\"\nalt=\"Add SSL Certificate to Load Balancer Screenshot\"\nclass=\"gallery-image\"\ndata-flex-grow=\"78\"\ndata-flex-basis=\"188px\"\n><\/p>\n<p>Once the certificate is created in the system, you can assign it to the load balancer and GCP will ask if you want to delete your unassigned certificate. My suggestion is that only delete it after you can confirm the new certificate is working without any issue.<\/p>\n<p>Similar to App Engine, the new certificate will only take effect after about 10 minutes or so. Verify it with all major browsers to make sure they like it.<\/p>\n<p>In my experience missing the certificate chain can cause issues with some browsers but Chrome might not mind it.<\/p>\n<h2 id=\"conclusion\">Conclusion\n<\/h2><p>Enabling SSL certificates in GCP load balancer and App Engine is fairly simple if you know about the process. Once you know how to navigate through the GCP UI to do it, it\u2019s fairly straightforward.<\/p>\n<p>Again, I think GCP should have a centralised place to manage all the certificates so you can just pick them up in the different products. That will save some effort.<\/p>\n<p>Leave a comment if you have questions on install SSL certificate on GCP.<\/p>"},{"title":"How to Use Docker for Go App Engine Development and Deploy to Standard Environment","link":"https:\/\/chaoming.li\/blog\/how-to-use-docker-for-go-app-engine-development-and-deploy-to-standard-environment\/","pubDate":"Mon, 17 Sep 2018 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/how-to-use-docker-for-go-app-engine-development-and-deploy-to-standard-environment\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/how-to-use-docker-for-go-app-engine-development-and-deploy-to-standard-environment\/43219741-fdfef302-8ffc-11e8-96f4-4a2ab2ce136b_orig-500x250.png\" alt=\"Featured image of post How to Use Docker for Go App Engine Development and Deploy to Standard Environment\" \/><p>App Engine is great but if you are using Docker, you can only use Flexible Environment which is not what I want as I would love to continue to use Standard Environment. Here is <a class=\"link\" href=\"https:\/\/cloud.google.com\/appengine\/docs\/the-appengine-environments\" target=\"_blank\" rel=\"noopener\"\n>a comparison of the two environments<\/a>.<\/p>\n<p>My goal is to use Docker for development only. When the app is deployed, it will still be deployed to App Engine Standard Environment as usual. After some researches and trial, I found a way to do so.<\/p>\n<h2 id=\"create-app-engine-development-docker\">Create App Engine Development Docker\n<\/h2><p>After I asked on <a class=\"link\" href=\"https:\/\/stackoverflow.com\/questions\/53719388\/is-it-possible-to-use-docker-as-dev-enviornment-for-golang-app-engine-standard-e\/53722860\" target=\"_blank\" rel=\"noopener\"\n>Stackoverflow<\/a>, and thanks to <a class=\"link\" href=\"https:\/\/stackoverflow.com\/users\/8456296\/dhauptman\" target=\"_blank\" rel=\"noopener\"\n>dhauptman<\/a> giving me the direction, I was able to create the Dockerfile that can spin up a docker container with Google Cloud SDK and run Go App Engine development environment. Here is the Dockerfile:<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\"> 1\n<\/span><span class=\"lnt\"> 2\n<\/span><span class=\"lnt\"> 3\n<\/span><span class=\"lnt\"> 4\n<\/span><span class=\"lnt\"> 5\n<\/span><span class=\"lnt\"> 6\n<\/span><span class=\"lnt\"> 7\n<\/span><span class=\"lnt\"> 8\n<\/span><span class=\"lnt\"> 9\n<\/span><span class=\"lnt\">10\n<\/span><span class=\"lnt\">11\n<\/span><span class=\"lnt\">12\n<\/span><span class=\"lnt\">13\n<\/span><span class=\"lnt\">14\n<\/span><span class=\"lnt\">15\n<\/span><span class=\"lnt\">16\n<\/span><span class=\"lnt\">17\n<\/span><span class=\"lnt\">18\n<\/span><span class=\"lnt\">19\n<\/span><span class=\"lnt\">20\n<\/span><span class=\"lnt\">21\n<\/span><span class=\"lnt\">22\n<\/span><span class=\"lnt\">23\n<\/span><span class=\"lnt\">24\n<\/span><span class=\"lnt\">25\n<\/span><span class=\"lnt\">26\n<\/span><span class=\"lnt\">27\n<\/span><span class=\"lnt\">28\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-docker\" data-lang=\"docker\"><span class=\"line\"><span class=\"cl\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"s\">golang<\/span><span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c\"># Install the Google Cloud SDK.<\/span><span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">RUN<\/span> <span class=\"nb\">echo<\/span> <span class=\"s2\">&#34;deb http:\/\/packages.cloud.google.com\/apt cloud-sdk-jessie main&#34;<\/span> <span class=\"p\">|<\/span> tee \/etc\/apt\/sources.list.d\/google-cloud-sdk.list<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">RUN<\/span> curl https:\/\/packages.cloud.google.com\/apt\/doc\/apt-key.gpg <span class=\"p\">|<\/span> apt-key add -<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">RUN<\/span> apt-get update <span class=\"o\">&amp;&amp;<\/span> apt-get install google-cloud-sdk google-cloud-sdk-app-engine-python<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-app-engine-python-extras<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-app-engine-java<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-app-engine-go<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-datalab<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-datastore-emulator<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-pubsub-emulator<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-cbt<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-cloud-build-local<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> google-cloud-sdk-bigtable-emulator<span class=\"se\">\\\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"> kubectl -y<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"c\"># Install go packages your app needs<\/span><span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">RUN<\/span> go get github.com\/go-sql-driver\/mysql<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">RUN<\/span> go get google.golang.org\/appengine<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">RUN<\/span> go get google.golang.org\/appengine\/log<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">RUN<\/span> go get google.golang.org\/appengine\/memcache<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">RUN<\/span> go get google.golang.org\/appengine\/taskqueue<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">COPY<\/span> .\/go_app \/go\/src\/app<span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">EXPOSE<\/span><span class=\"w\"> <\/span><span class=\"s\">8080<\/span><span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">EXPOSE<\/span><span class=\"w\"> <\/span><span class=\"s\">8000<\/span><span class=\"err\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">CMD<\/span> <span class=\"p\">[<\/span><span class=\"s2\">&#34;dev_appserver.py&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;--host=0.0.0.0&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;--admin_host=0.0.0.0&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;--storage_path=~\/appengine_storage&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;--blobstore_path=~\/appengine_blobstore&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;--datastore_path=~\/appengine_datastore&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;--log_level=debug&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;--enable_host_checking=false&#34;<\/span><span class=\"p\">,<\/span> <span class=\"s2\">&#34;\/go\/src\/app\/app.yaml&#34;<\/span><span class=\"p\">]<\/span><span class=\"err\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>I assume you have basic Docker knowledge so I am not going to explain the Dockerfile step by step here. One thing that cost me some time was the CMD. You must have \u201c\u2013host=0.0.0.0\u201d and \u201c\u2013admin_host=0.0.0.0\u201d in your CMD otherwise you will not be able to access your app engine development instances from your local environment.<\/p>\n<p>To save the trouble of typing the docker run parameters. I also created a docker-compose.yaml file. But it\u2019s not really necessary if you have only one Docker container to run.<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\"> 1\n<\/span><span class=\"lnt\"> 2\n<\/span><span class=\"lnt\"> 3\n<\/span><span class=\"lnt\"> 4\n<\/span><span class=\"lnt\"> 5\n<\/span><span class=\"lnt\"> 6\n<\/span><span class=\"lnt\"> 7\n<\/span><span class=\"lnt\"> 8\n<\/span><span class=\"lnt\"> 9\n<\/span><span class=\"lnt\">10\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-yaml\" data-lang=\"yaml\"><span class=\"line\"><span class=\"cl\"><span class=\"nt\">version<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;3&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nt\">services<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">go-app-engine<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">build<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"l\">.<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">volumes<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span>- <span class=\"l\">.\/go_app \/go\/src\/app<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">ports<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span>- <span class=\"m\">8080<\/span><span class=\"p\">:<\/span><span class=\"m\">8080<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span>- <span class=\"m\">8000<\/span><span class=\"p\">:<\/span><span class=\"m\">8000<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>Now just need to use the following command to get the Go App Engine development environment up and running.<\/p>\n<p><code>docker-compose up<\/code><\/p>\n<h2 id=\"deploy-to-app-engine-standard-environment\">Deploy to App Engine Standard Environment\n<\/h2><p>I don\u2019t want to deploy Docker containers to App Engine because that allows using Flexible Environment only. I will need to find a different way to deploy to Standard Environment. Google Cloud Build is perfect for the deployment task. It can deploy code from Github repository triggered by Git push.<\/p>\n<p>The solution is fairly straightforward. First, push the Go source code to Github. Git push will trigger Cloud Build to run cloudbuild.yaml and go through the steps to get the Go packages and to run the gcloud deploy command.<\/p>\n<p>To do this, make sure your App Engine admin API is enabled and the App Engine is created in your project. You will also need to grant App Engine deployer and server admin permissions to the cloud build service account as below screenshot.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/how-to-use-docker-for-go-app-engine-development-and-deploy-to-standard-environment\/grant-permission_orig.png\"\nwidth=\"530\"\nheight=\"300\"\nsrcset=\"https:\/\/chaoming.li\/blog\/how-to-use-docker-for-go-app-engine-development-and-deploy-to-standard-environment\/grant-permission_orig_hu_a933d565fd99ded7.png 480w, https:\/\/chaoming.li\/blog\/how-to-use-docker-for-go-app-engine-development-and-deploy-to-standard-environment\/grant-permission_orig_hu_1ac1f3347ce67a36.png 1024w\"\nloading=\"lazy\"\nalt=\"Permissions for Cloud Build to Deploy to App Engine\"\nclass=\"gallery-image\"\ndata-flex-grow=\"176\"\ndata-flex-basis=\"424px\"\n><\/p>\n<p>Here is the cloudbuild.yaml file. Similar to the Dockerfile, it specifics the steps to install the Go packages and then run the gcloud deploy command.<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\"> 1\n<\/span><span class=\"lnt\"> 2\n<\/span><span class=\"lnt\"> 3\n<\/span><span class=\"lnt\"> 4\n<\/span><span class=\"lnt\"> 5\n<\/span><span class=\"lnt\"> 6\n<\/span><span class=\"lnt\"> 7\n<\/span><span class=\"lnt\"> 8\n<\/span><span class=\"lnt\"> 9\n<\/span><span class=\"lnt\">10\n<\/span><span class=\"lnt\">11\n<\/span><span class=\"lnt\">12\n<\/span><span class=\"lnt\">13\n<\/span><span class=\"lnt\">14\n<\/span><span class=\"lnt\">15\n<\/span><span class=\"lnt\">16\n<\/span><span class=\"lnt\">17\n<\/span><span class=\"lnt\">18\n<\/span><span class=\"lnt\">19\n<\/span><span class=\"lnt\">20\n<\/span><span class=\"lnt\">21\n<\/span><span class=\"lnt\">22\n<\/span><span class=\"lnt\">23\n<\/span><span class=\"lnt\">24\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-yaml\" data-lang=\"yaml\"><span class=\"line\"><span class=\"cl\"><span class=\"nt\">steps<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\">- <span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gcr.io\/cloud-builders\/go&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">args<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;get&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;github.com\/go-sql-driver\/mysql&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">env<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;GOPATH=go&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\">- <span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gcr.io\/cloud-builders\/go&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">args<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;get&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;google.golang.org\/appengine&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">env<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;GOPATH=go&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\">- <span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gcr.io\/cloud-builders\/go&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">args<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;get&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;google.golang.org\/appengine\/log&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">env<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;GOPATH=go&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\">- <span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gcr.io\/cloud-builders\/go&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">args<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;get&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;google.golang.org\/appengine\/memcache&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">env<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;GOPATH=go&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\">- <span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gcr.io\/cloud-builders\/go&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">args<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;get&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;google.golang.org\/appengine\/taskqueue&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">env<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;GOPATH=go&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\">- <span class=\"nt\">name<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gcr.io\/cloud-builders\/gcloud&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">args<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;app&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;deploy&#39;<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;go_app\/&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">env<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;GOPATH=go&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"nt\">artifacts<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">objects<\/span><span class=\"p\">:<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">location<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;gs:\/\/bucket-nam\/&#39;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"nt\">paths<\/span><span class=\"p\">:<\/span><span class=\"w\"> <\/span><span class=\"p\">[<\/span><span class=\"s1\">&#39;go_app&#39;<\/span><span class=\"p\">]<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>Now you just need to create the Cloud Build trigger to be triggered by Git push and run the cloudbuild.yaml to package the Go source code and deploy it to App Engine Standard Environment.<\/p>\n<h2 id=\"conclusion\">Conclusion\n<\/h2><p>This approach is great when you need to replicate the development environment without much overhead. Including the Dockerfile in the Git repository will allow new developers to set up a development environment very fast as long as Git and Docker are installed.<\/p>\n<p>Deployment is fully automatic when the code is pushed to the Github repository. I am not going to talk about how to manage and review code before deployment but you will need to have quality control on the code being pushed into the Github repository anyway.<\/p>\n<p>Let me know what you think or if you have any questions.<\/p>"},{"title":"Joining Partitioned Tables to Create Views in BigQuery","link":"https:\/\/chaoming.li\/blog\/joining-partitioned-tables-to-create-views-in-bigquery\/","pubDate":"Fri, 30 Mar 2018 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/joining-partitioned-tables-to-create-views-in-bigquery\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/joining-partitioned-tables-to-create-views-in-bigquery\/bigquery-500x250.png\" alt=\"Featured image of post Joining Partitioned Tables to Create Views in BigQuery\" \/><p>BigQuery date partitioned tables can limit the data scan by partitions to help keep the query cost low and improve query performance. However, Google\u2019s documents do not give much clue about how to use partitioned tables to create views that support partition queries. I did some research and found the way to do it, even with views that are created from joining partitioned tables.<\/p>\n<p>To expose the partitioning pseudo-column, create a query similar to this one:<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\">1\n<\/span><span class=\"lnt\">2\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-sql\" data-lang=\"sql\"><span class=\"line\"><span class=\"cl\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"k\">EXTRACT<\/span><span class=\"p\">(<\/span><span class=\"nb\">DATE<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">_PARTITIONTIME<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"nb\">date<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">partitioned<\/span><span class=\"o\">-<\/span><span class=\"k\">table<\/span><span class=\"p\">;<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>If you save the query as a view, you can limit the query partitions by using the date column in your <code>WHERE<\/code> clause.<\/p>\n<p>The above example is probably too simple for any actual query. We often create views because we have complex queries that join multiple tables. In that case, to make the views support partitions, it is just as simple as creating multiple date columns as the above example and making sure your query of the view contains a <code>WHERE<\/code> clause that limits the search of these date columns. Here is an example of joining two partition tables:<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\">1\n<\/span><span class=\"lnt\">2\n<\/span><span class=\"lnt\">3\n<\/span><span class=\"lnt\">4\n<\/span><span class=\"lnt\">5\n<\/span><span class=\"lnt\">6\n<\/span><span class=\"lnt\">7\n<\/span><span class=\"lnt\">8\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-sql\" data-lang=\"sql\"><span class=\"line\"><span class=\"cl\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"k\">EXTRACT<\/span><span class=\"p\">(<\/span><span class=\"nb\">DATE<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">_PARTITIONTIME<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">date1<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">partitioned<\/span><span class=\"o\">-<\/span><span class=\"n\">table1<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"n\">t1<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">LEFT<\/span><span class=\"w\"> <\/span><span class=\"k\">JOIN<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"k\">EXTRACT<\/span><span class=\"p\">(<\/span><span class=\"nb\">DATE<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">_PARTITIONTIME<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">date2<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">partitioned<\/span><span class=\"o\">-<\/span><span class=\"n\">table1<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"n\">t2<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">ON<\/span><span class=\"w\"> <\/span><span class=\"n\">t1<\/span><span class=\"p\">.<\/span><span class=\"k\">key<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"n\">t2<\/span><span class=\"p\">.<\/span><span class=\"k\">key<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"p\">);<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>When you query this view, and you want to limit the query to the data of <code>2018-03-20<\/code>, you can do this:<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\">1\n<\/span><span class=\"lnt\">2\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-sql\" data-lang=\"sql\"><span class=\"line\"><span class=\"cl\"><span class=\"k\">SELECT<\/span><span class=\"w\"> <\/span><span class=\"o\">*<\/span><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"n\">partitioned<\/span><span class=\"o\">-<\/span><span class=\"k\">view<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">WHERE<\/span><span class=\"w\"> <\/span><span class=\"n\">date1<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;2018-03-20&#39;<\/span><span class=\"w\"> <\/span><span class=\"k\">AND<\/span><span class=\"w\"> <\/span><span class=\"n\">date2<\/span><span class=\"w\"> <\/span><span class=\"o\">=<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;2018-03-20&#39;<\/span><span class=\"p\">;<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>I wish I could combine the different date columns as one, so I tried to join the date columns as keys in the ON statement, but that doesn\u2019t help. It seems you always have to have one date column for each partitioned table in the query, that\u2019s a little bit annoying but it will help lower the costs.<\/p>"},{"title":"Build Your Own Web Analytics Platform in 25 Minutes","link":"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/","pubDate":"Tue, 20 Feb 2018 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-19-46-41_orig-768x427.png\" alt=\"Featured image of post Build Your Own Web Analytics Platform in 25 Minutes\" \/><p>I gave a presentation in MeasureCamp Melbourne, and the topic was \u201cBuild Your Own Web Analytics Platform in 25 Minutes\u201d. In MeasureCamp, each session has 25 minutes of presentation time, that\u2019s where the 25 minutes on the topic from. What inspired me to do this talk was a conversation in Measure Slack about sending Google Analytics data into your own database. And I think it is an interesting thing to do to build a web analytics platform from scratch without spending a lot of time and money.<\/p>\n<p>Building your own web analytics platform can be much simpler than you would think. In this presentation, there is pretty much no coding knowledge required. If you are familiar with AWS S3 and CloudFront, it is very simple. If not, don\u2019t worry, AWS has a really good interface so you just need to click through a series of buttons. Below is the high-level architecture of the platform. Option #1 was something I built in my previous job for production support. It was not based on any cloud so it was able to store personally identifiable information (PII) and the production support team used it to help to recreate scenarios for debugging. Option #2 is the solution in this presentation, which uses AWS cloud so all the infrastructure can be set up in minutes. You will need to have an AWS account to do it. You can <a class=\"link\" href=\"https:\/\/aws.amazon.com\/free\/\" target=\"_blank\" rel=\"noopener\"\n>sign up for a free account here<\/a>.<\/p>\n<p>The whole process includes 8 steps. I will go through all the 8 steps here.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-19-46-41_orig-768x427.png\"\nwidth=\"768\"\nheight=\"427\"\nsrcset=\"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-19-46-41_orig-768x427_hu_61e15fe6c9dac6e2.png 480w, https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-19-46-41_orig-768x427_hu_55e20ff45b14428b.png 1024w\"\nloading=\"lazy\"\nalt=\"Simple Web Analytics System Architecture\"\nclass=\"gallery-image\"\ndata-flex-grow=\"179\"\ndata-flex-basis=\"431px\"\n><\/p>\n<h2 id=\"step-1-create-an-s3-bucket-to-store-log-files\">Step 1: Create an S3 bucket to store log files\n<\/h2><p>First of all, you will need an S3 bucket to store all the log files. AWS S3 is a storage service, where you can store a lot of files in the cloud with pretty low costs. S3 buckets are like folders, you can store a group of files within.<\/p>\n<p>Login to your AWS account, click on S3 in the product list menu under the Storage section and click on Create Bucket button. Give a name to the bucket, something like \u201cmy-web-analytics-logs\u201d, and click Next all the way through. You have your S3 bucket created now.<\/p>\n<h2 id=\"step-2-create-an-s3-bucket-to-store-the-tracking-pixel-image\">Step 2: Create an S3 bucket to store the tracking pixel image\n<\/h2><p>Most web analytics solutions are using an invisible pixel image to track data. They attach the information to be tracked in the pixel URL as query parameters. So we need an S3 bucket to host this pixel file. Just repeat the process of step 1, but name the bucket \u201cmy-web-analytics-pixel\u201d. Upload the pixel file to the bucket. You can grab the pixel file from <a class=\"link\" href=\"https:\/\/s3.amazonaws.com\/cli-tracking-demo-pixel\/pixel.gif\" target=\"_blank\" rel=\"noopener\"\n>https:\/\/s3.amazonaws.com\/cli-tracking-demo-pixel\/pixel.gif<\/a><\/p>\n<p>You will need to make the pixel file public so it can be accessed from the Internet. To do so, go to the pixel bucket and tick the checkbox next to the file name, click More and select Make Public.<\/p>\n<h2 id=\"step-3-create-a-cloudfront-distribution-and-enable-logging\">Step 3: Create a CloudFront distribution and enable logging\n<\/h2><p>Now, we have our pixel file in the cloud, and we are ready to store the log files. CloudFront is the CDN AWS offers. It is fast and cheap to run. The reason we want to use CloudFront here is to be able to serve the pixel file very fast, so it doesn\u2019t impact the page performance much, and log the data at the same time. Go to CloudFront, click on Create Distribution button, and click on Get Started under the Web section to go to the form to create a distribution.<\/p>\n<p>When you create the CloudFront distribution, you will need to enable logging and point to log files to the S3 bucket we created in step 1.<\/p>\n<p>You will also need to point the origin to the S3 bucket containing the pixel file so your CloudFront distribution can serve the pixel file. You can leave all the other options as default.<\/p>\n<p>After these 3 steps are completed, you are ready to take in whatever data and store it. We should start to work on the part to generate data.<\/p>\n<h2 id=\"step-4-create-a-visitor-id-via-gtm\">Step 4: Create a visitor ID via GTM\n<\/h2><p>To track the same visitors across a period, you need a cookie to store a unique random visitor ID. In Google Tag Manager (GTM), go to Variables, and create a custom Javascript variable and name it \u201cVisitor ID\u201d. Copy and paste the code below to the variable. This piece of code will check if the visitor already has a visitor ID cookie \u201cv_id\u201d. If yes, it will read it, refresh the cookie for another 2 years, and return the visitor ID value. Otherwise, it will create a random ID and save it to \u201cv_id\u201d cookie for 2 years, and return the newly created visitor ID value. We will use this value in the next step.<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\"> 1\n<\/span><span class=\"lnt\"> 2\n<\/span><span class=\"lnt\"> 3\n<\/span><span class=\"lnt\"> 4\n<\/span><span class=\"lnt\"> 5\n<\/span><span class=\"lnt\"> 6\n<\/span><span class=\"lnt\"> 7\n<\/span><span class=\"lnt\"> 8\n<\/span><span class=\"lnt\"> 9\n<\/span><span class=\"lnt\">10\n<\/span><span class=\"lnt\">11\n<\/span><span class=\"lnt\">12\n<\/span><span class=\"lnt\">13\n<\/span><span class=\"lnt\">14\n<\/span><span class=\"lnt\">15\n<\/span><span class=\"lnt\">16\n<\/span><span class=\"lnt\">17\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-jsx\" data-lang=\"jsx\"><span class=\"line\"><span class=\"cl\"><span class=\"kd\">function<\/span><span class=\"p\">(){<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"kd\">var<\/span> <span class=\"nx\">readCookie<\/span> <span class=\"o\">=<\/span> <span class=\"kd\">function<\/span><span class=\"p\">(<\/span><span class=\"nx\">key<\/span><span class=\"p\">){<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"kd\">var<\/span> <span class=\"nx\">result<\/span><span class=\"p\">;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">return<\/span> <span class=\"p\">(<\/span><span class=\"nx\">result<\/span> <span class=\"o\">=<\/span> <span class=\"k\">new<\/span> <span class=\"nb\">RegExp<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;(?:^|; )&#39;<\/span> <span class=\"o\">+<\/span> <span class=\"nb\">encodeURIComponent<\/span><span class=\"p\">(<\/span><span class=\"nx\">key<\/span><span class=\"p\">)<\/span> <span class=\"o\">+<\/span> <span class=\"s1\">&#39;=([^;]*)&#39;<\/span><span class=\"p\">).<\/span><span class=\"nx\">exec<\/span><span class=\"p\">(<\/span><span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">cookie<\/span><span class=\"p\">))<\/span> <span class=\"o\">?<\/span> <span class=\"p\">(<\/span><span class=\"nx\">result<\/span><span class=\"p\">[<\/span><span class=\"mi\">1<\/span><span class=\"p\">])<\/span> <span class=\"o\">:<\/span> <span class=\"kc\">null<\/span><span class=\"p\">;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"p\">}<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"kd\">var<\/span> <span class=\"nx\">writeCookie<\/span> <span class=\"o\">=<\/span> <span class=\"kd\">function<\/span><span class=\"p\">(<\/span><span class=\"nx\">name<\/span><span class=\"p\">,<\/span> <span class=\"nx\">value<\/span><span class=\"p\">,<\/span> <span class=\"nx\">domain<\/span><span class=\"p\">,<\/span> <span class=\"nx\">expire<\/span><span class=\"p\">){<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">document<\/span><span class=\"p\">.<\/span><span class=\"nx\">cookie<\/span> <span class=\"o\">=<\/span> <span class=\"nx\">name<\/span><span class=\"o\">+<\/span><span class=\"s2\">&#34;=&#34;<\/span><span class=\"o\">+<\/span><span class=\"nx\">value<\/span><span class=\"o\">+<\/span><span class=\"s2\">&#34;;domain=.&#34;<\/span><span class=\"o\">+<\/span><span class=\"nx\">domain<\/span><span class=\"o\">+<\/span><span class=\"s2\">&#34;;path=\/;expires=&#34;<\/span><span class=\"o\">+<\/span><span class=\"nx\">expire<\/span><span class=\"p\">;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"p\">}<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"kd\">var<\/span> <span class=\"nx\">visitorId<\/span> <span class=\"o\">=<\/span> <span class=\"nx\">readCookie<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;v_id&#39;<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"kd\">var<\/span> <span class=\"nx\">date<\/span> <span class=\"o\">=<\/span> <span class=\"k\">new<\/span> <span class=\"nb\">Date<\/span><span class=\"p\">();<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">if<\/span><span class=\"p\">(<\/span><span class=\"nx\">visitorId<\/span> <span class=\"o\">===<\/span> <span class=\"kc\">null<\/span><span class=\"p\">){<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nx\">visitorId<\/span> <span class=\"o\">=<\/span> <span class=\"nx\">date<\/span><span class=\"p\">.<\/span><span class=\"nx\">getTime<\/span><span class=\"p\">().<\/span><span class=\"nx\">toString<\/span><span class=\"p\">(<\/span><span class=\"mi\">16<\/span><span class=\"p\">)<\/span> <span class=\"o\">+<\/span> <span class=\"p\">(<\/span><span class=\"nb\">Math<\/span><span class=\"p\">.<\/span><span class=\"nx\">floor<\/span><span class=\"p\">(<\/span><span class=\"nb\">Math<\/span><span class=\"p\">.<\/span><span class=\"nx\">random<\/span><span class=\"p\">()<\/span> <span class=\"o\">*<\/span> <span class=\"p\">(<\/span><span class=\"mi\">999999<\/span> <span class=\"o\">-<\/span> <span class=\"mi\">100000<\/span><span class=\"p\">)<\/span> <span class=\"o\">+<\/span> <span class=\"mi\">100000<\/span><span class=\"p\">)).<\/span><span class=\"nx\">toString<\/span><span class=\"p\">(<\/span><span class=\"mi\">16<\/span><span class=\"p\">);<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"p\">}<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nx\">date<\/span><span class=\"p\">.<\/span><span class=\"nx\">setTime<\/span><span class=\"p\">(<\/span><span class=\"nx\">date<\/span><span class=\"p\">.<\/span><span class=\"nx\">getTime<\/span><span class=\"p\">()<\/span> <span class=\"o\">+<\/span> <span class=\"p\">(<\/span><span class=\"mi\">2<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">365<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">24<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">60<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">60<\/span> <span class=\"o\">*<\/span> <span class=\"mi\">1000<\/span><span class=\"p\">));<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nx\">writeCookie<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;v_id&#39;<\/span><span class=\"p\">,<\/span> <span class=\"nx\">visitorId<\/span><span class=\"p\">,<\/span> <span class=\"nx\">location<\/span><span class=\"p\">.<\/span><span class=\"nx\">hostname<\/span><span class=\"p\">,<\/span> <span class=\"k\">new<\/span> <span class=\"nb\">Date<\/span><span class=\"p\">(<\/span><span class=\"nx\">date<\/span><span class=\"p\">).<\/span><span class=\"nx\">toUTCString<\/span><span class=\"p\">());<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">return<\/span> <span class=\"nx\">visitorId<\/span><span class=\"p\">;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"p\">}<\/span>\n<\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><h2 id=\"step-5-create-an-image-tag-in-gtm-and-attach-data-in-the-url-query-parameters\">Step 5: Create an image tag in GTM and attach data in the URL query parameters\n<\/h2><p>We are going to use the pixel file in this step. Create an image tag in GTM. In the URL field, put in \u201chttps:\/\/your-cloud-front-distribution.cloudfront.net\/pixel.gif?vid={{Visitor ID}}&amp;pageUrl={{Page URL}}\u201d, set the trigger as All Pages. <code>{{Visitor ID}}<\/code> is the variable created in the previous step, and I attached the page URL in the query parameters in this example but you can attach whatever variables in your GTM as key-value pairs in the same way.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-20-29-52_orig-768x337.png\"\nwidth=\"768\"\nheight=\"337\"\nsrcset=\"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-20-29-52_orig-768x337_hu_36f249eb91270f4d.png 480w, https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-20-29-52_orig-768x337_hu_58a7a0fd217a32c3.png 1024w\"\nloading=\"lazy\"\nalt=\"Google Tag Manager Custom Tag\"\nclass=\"gallery-image\"\ndata-flex-grow=\"227\"\ndata-flex-basis=\"546px\"\n><\/p>\n<h2 id=\"step-6-publish-gtm\">Step 6: Publish GTM\n<\/h2><p>By now, all the data collection and storage is set up. Once you publish GTM to your websites, data will start to flow in soon. The way it is set up in this demo doesn\u2019t support real-time data streaming, so you will need to wait a few minutes to have your first log file deposited in the S3 bucket.<\/p>\n<h2 id=\"step-7-create-an-athena-table-schema\">Step 7: Create an Athena table schema\n<\/h2><p>After the data files are in your S3 bucket, you can start to query the data. AWS Athena is probably the fastest way to do so if you like to use SQL. I prefer to create a table manually in Athena than using the crawler feature because I found the crawler is not always working properly in my experience.<\/p>\n<p>To manually create a table in Athena, you will need to point the table to your log file s3 bucket and define all the fields in the table. You can find <a class=\"link\" href=\"https:\/\/docs.aws.amazon.com\/AmazonCloudFront\/latest\/DeveloperGuide\/AccessLogs.html\" target=\"_blank\" rel=\"noopener\"\n>the CloudFront log file format here<\/a>. There are 25 fields, so take your time to go through them one by one. In my demo, I only created the first 12 fields because that\u2019s good enough for the demo, and the 12th field is the one stores the query parameter data. Here is a screenshot of the fields in my table.<\/p>\n<p><img src=\"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-20-39-40_orig-768x381.png\"\nwidth=\"768\"\nheight=\"381\"\nsrcset=\"https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-20-39-40_orig-768x381_hu_c1d6b3a6c9f16bdb.png 480w, https:\/\/chaoming.li\/blog\/build-your-own-web-analytics-platform-in-25-minutes\/screenshot-2018-02-24-20-39-40_orig-768x381_hu_88d0cd5e2a30697e.png 1024w\"\nloading=\"lazy\"\nalt=\"AWS Athena Data Schema\"\nclass=\"gallery-image\"\ndata-flex-grow=\"201\"\ndata-flex-basis=\"483px\"\n><\/p>\n<h2 id=\"step-8-check-your-data\"><strong>S<\/strong>tep 8: Check your data\n<\/h2><p>You can use \u201cselect * from database_name.table_name\u201d to inspect all the data you have in the log files. Spend some time learning your data, you have the IP address, the user agent string, and the data you send through from the pixel.<\/p>\n<p>Here is the SQL to count visitors by page URLs and sort the results in descending order. The example uses the <code>url_extract_parameter<\/code> function to get the value out of the query parameters by name. This function only works for full URLs, so I have to concatenate some fields to recreate the full URLs to allow the function to work properly,<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\"> 1\n<\/span><span class=\"lnt\"> 2\n<\/span><span class=\"lnt\"> 3\n<\/span><span class=\"lnt\"> 4\n<\/span><span class=\"lnt\"> 5\n<\/span><span class=\"lnt\"> 6\n<\/span><span class=\"lnt\"> 7\n<\/span><span class=\"lnt\"> 8\n<\/span><span class=\"lnt\"> 9\n<\/span><span class=\"lnt\">10\n<\/span><span class=\"lnt\">11\n<\/span><span class=\"lnt\">12\n<\/span><span class=\"lnt\">13\n<\/span><span class=\"lnt\">14\n<\/span><span class=\"lnt\">15\n<\/span><span class=\"lnt\">16\n<\/span><span class=\"lnt\">17\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-sql\" data-lang=\"sql\"><span class=\"line\"><span class=\"cl\"><span class=\"k\">SELECT<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">COUNT<\/span><span class=\"p\">(<\/span><span class=\"k\">DISTINCT<\/span><span class=\"w\"> <\/span><span class=\"n\">VisitorID<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">Visitors<\/span><span class=\"p\">,<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"n\">PageUrl<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">SELECT<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"n\">url_extract_parameter<\/span><span class=\"p\">(<\/span><span class=\"n\">RequestURL<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;vid&#39;<\/span><span class=\"p\">)<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">VisitorID<\/span><span class=\"p\">,<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"n\">url_decode<\/span><span class=\"p\">(<\/span><span class=\"n\">url_extract_parameter<\/span><span class=\"p\">(<\/span><span class=\"n\">RequestURL<\/span><span class=\"p\">,<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;url&#39;<\/span><span class=\"p\">))<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">PageUrl<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\"> <\/span><span class=\"p\">(<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">SELECT<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"s1\">&#39;https:\/\/&#39;<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"s2\">&#34;cs-host&#34;<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"s2\">&#34;cs-uri-stem&#34;<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"s1\">&#39;?&#39;<\/span><span class=\"w\"> <\/span><span class=\"o\">||<\/span><span class=\"w\"> <\/span><span class=\"s2\">&#34;cs-uri-query&#34;<\/span><span class=\"w\"> <\/span><span class=\"k\">AS<\/span><span class=\"w\"> <\/span><span class=\"n\">RequestURL<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"k\">FROM<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"s2\">&#34;webanalytics&#34;<\/span><span class=\"p\">.<\/span><span class=\"s2\">&#34;logs&#34;<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"p\">)<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"p\">)<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">GROUP<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"w\"> <\/span><span class=\"n\">PageUrl<\/span><span class=\"w\">\n<\/span><\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">ORDER<\/span><span class=\"w\"> <\/span><span class=\"k\">BY<\/span><span class=\"w\"> <\/span><span class=\"n\">Visitors<\/span><span class=\"w\"> <\/span><span class=\"k\">DESC<\/span><span class=\"w\">\n<\/span><\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><h2 id=\"conclusion\">Conclusion\n<\/h2><p>By now, you have a very simple web analytics platform that can take in whatever data you want to track as long as you have it in your GTM variables. This is just a super-simplified demo of how web analytics platforms work in principle. There are a lot of missing parts such as cross-domain tracking, data enrichment, data visualisation, etc. However, it is also very flexible because it is not limited by any platform. You have the freedom to design what to track.<\/p>\n<p>I hope this article helps you understand how web analytics platform collects data at a high level, and feel free to extend it to other usages like tracking non-web data.<\/p>"},{"title":"Python Voice Assistant Source Code","link":"https:\/\/chaoming.li\/blog\/python-voice-assistant-source-code\/","pubDate":"Mon, 15 Jan 2018 00:00:00 +0000","guid":"https:\/\/chaoming.li\/blog\/python-voice-assistant-source-code\/","description":"<img src=\"https:\/\/chaoming.li\/blog\/python-voice-assistant-source-code\/pytho-voice-assistant.png\" alt=\"Featured image of post Python Voice Assistant Source Code\" \/><p>Back in mid-2015, I created my first and only Python script so far. The script was a voice assistant. I thought I lost the source code, but it\u2019s been in my Dropbox all the time and was named with a weird project name so I didn\u2019t realise that\u2019s it. I just didn\u2019t try to look for it hard enough.<\/p>\n<p>Here are a couple of videos I took back in 2015 when it was built. The script was running on a Raspberry Pi 3 with a USB sound card connecting to a microphone and a speaker.<\/p>\n<div class=\"video-wrapper\">\n<iframe loading=\"lazy\"\nsrc=\"https:\/\/www.youtube.com\/embed\/mphr9-64irA\"\nallowfullscreen\ntitle=\"YouTube Video\"\n>\n<\/iframe>\n<\/div>\n<div class=\"video-wrapper\">\n<iframe loading=\"lazy\"\nsrc=\"https:\/\/www.youtube.com\/embed\/4IHVUHfimCw\"\nallowfullscreen\ntitle=\"YouTube Video\"\n>\n<\/iframe>\n<\/div>\n<p>You will notice that it was pretty slow in response. That\u2019s probably because I couldn\u2019t figure out how to lower the sound input sample rate. The rate was 44.1KHz. I think 8KHz is good enough and it can cut down the data size significantly.<\/p>\n<p>How the script works is fairly simple. Get the voice data from the USB sound card, and send it to Google Speech Recognition API if the sound goes above a threshold. Get the text back from Google API and decide what to do. It can only handle two kinds of commands. If you say \u201c(.<em>)(tell|say)(.<\/em>) about (.<em>)\u201d, it will call Wikipedia for the last \u201c(.<\/em>)\u201d which is supposed to be a name or a thing. Otherwise, it will call WolframAlpha to try to get an answer to your question. Once it gets the text for the response, it responds through the speaker.<\/p>\n<p>The most challenging part of this project was not coding. The original plan was to use a Bluetooth speaker with microphone and I spent a long time trying to make the speaker work with Raspberry Pi but failed.<\/p>\n<p>Now, here is the source code. I can\u2019t guarantee it still works since it\u2019s over 2 years old. It might not even be the working version on my Raspberry Pi. But I hope this is helpful for anyone who is interesting in building a voice assistant.<\/p>\n<div class=\"highlight\"><div class=\"chroma\">\n<table class=\"lntable\"><tr><td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code><span class=\"lnt\"> 1\n<\/span><span class=\"lnt\"> 2\n<\/span><span class=\"lnt\"> 3\n<\/span><span class=\"lnt\"> 4\n<\/span><span class=\"lnt\"> 5\n<\/span><span class=\"lnt\"> 6\n<\/span><span class=\"lnt\"> 7\n<\/span><span class=\"lnt\"> 8\n<\/span><span class=\"lnt\"> 9\n<\/span><span class=\"lnt\">10\n<\/span><span class=\"lnt\">11\n<\/span><span class=\"lnt\">12\n<\/span><span class=\"lnt\">13\n<\/span><span class=\"lnt\">14\n<\/span><span class=\"lnt\">15\n<\/span><span class=\"lnt\">16\n<\/span><span class=\"lnt\">17\n<\/span><span class=\"lnt\">18\n<\/span><span class=\"lnt\">19\n<\/span><span class=\"lnt\">20\n<\/span><span class=\"lnt\">21\n<\/span><span class=\"lnt\">22\n<\/span><span class=\"lnt\">23\n<\/span><span class=\"lnt\">24\n<\/span><span class=\"lnt\">25\n<\/span><span class=\"lnt\">26\n<\/span><span class=\"lnt\">27\n<\/span><span class=\"lnt\">28\n<\/span><span class=\"lnt\">29\n<\/span><span class=\"lnt\">30\n<\/span><span class=\"lnt\">31\n<\/span><span class=\"lnt\">32\n<\/span><span class=\"lnt\">33\n<\/span><span class=\"lnt\">34\n<\/span><span class=\"lnt\">35\n<\/span><span class=\"lnt\">36\n<\/span><span class=\"lnt\">37\n<\/span><span class=\"lnt\">38\n<\/span><span class=\"lnt\">39\n<\/span><span class=\"lnt\">40\n<\/span><span class=\"lnt\">41\n<\/span><span class=\"lnt\">42\n<\/span><span class=\"lnt\">43\n<\/span><span class=\"lnt\">44\n<\/span><span class=\"lnt\">45\n<\/span><span class=\"lnt\">46\n<\/span><span class=\"lnt\">47\n<\/span><span class=\"lnt\">48\n<\/span><span class=\"lnt\">49\n<\/span><span class=\"lnt\">50\n<\/span><span class=\"lnt\">51\n<\/span><span class=\"lnt\">52\n<\/span><span class=\"lnt\">53\n<\/span><span class=\"lnt\">54\n<\/span><span class=\"lnt\">55\n<\/span><span class=\"lnt\">56\n<\/span><span class=\"lnt\">57\n<\/span><span class=\"lnt\">58\n<\/span><span class=\"lnt\">59\n<\/span><span class=\"lnt\">60\n<\/span><span class=\"lnt\">61\n<\/span><span class=\"lnt\">62\n<\/span><span class=\"lnt\">63\n<\/span><span class=\"lnt\">64\n<\/span><span class=\"lnt\">65\n<\/span><span class=\"lnt\">66\n<\/span><span class=\"lnt\">67\n<\/span><span class=\"lnt\">68\n<\/span><span class=\"lnt\">69\n<\/span><span class=\"lnt\">70\n<\/span><\/code><\/pre><\/td>\n<td class=\"lntd\">\n<pre tabindex=\"0\" class=\"chroma\"><code class=\"language-python\" data-lang=\"python\"><span class=\"line\"><span class=\"cl\"><span class=\"kn\">import<\/span> <span class=\"nn\">speech_recognition<\/span> <span class=\"k\">as<\/span> <span class=\"nn\">sr<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"kn\">import<\/span> <span class=\"nn\">time<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"kn\">import<\/span> <span class=\"nn\">pyttsx<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"kn\">import<\/span> <span class=\"nn\">threading<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"kn\">import<\/span> <span class=\"nn\">wikipedia<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"kn\">import<\/span> <span class=\"nn\">re<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"kn\">import<\/span> <span class=\"nn\">wolframalpha<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">def<\/span> <span class=\"nf\">listen<\/span><span class=\"p\">():<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">def<\/span> <span class=\"nf\">say<\/span><span class=\"p\">(<\/span><span class=\"n\">text<\/span><span class=\"p\">):<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">engine<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pyttsx<\/span><span class=\"o\">.<\/span><span class=\"n\">init<\/span><span class=\"p\">()<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">engine<\/span><span class=\"o\">.<\/span><span class=\"n\">setProperty<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;rate&#34;<\/span><span class=\"p\">,<\/span> <span class=\"mi\">150<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">engine<\/span><span class=\"o\">.<\/span><span class=\"n\">setProperty<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;volume&#34;<\/span><span class=\"p\">,<\/span> <span class=\"mf\">0.3<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">engine<\/span><span class=\"o\">.<\/span><span class=\"n\">say<\/span><span class=\"p\">(<\/span><span class=\"n\">text<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">engine<\/span><span class=\"o\">.<\/span><span class=\"n\">runAndWait<\/span><span class=\"p\">()<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">def<\/span> <span class=\"nf\">wiki<\/span><span class=\"p\">(<\/span><span class=\"n\">text<\/span><span class=\"p\">):<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">summary<\/span> <span class=\"o\">=<\/span> <span class=\"n\">wikipedia<\/span><span class=\"o\">.<\/span><span class=\"n\">summary<\/span><span class=\"p\">(<\/span><span class=\"n\">text<\/span><span class=\"p\">,<\/span> <span class=\"n\">sentences<\/span><span class=\"o\">=<\/span><span class=\"mi\">3<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">summary<\/span> <span class=\"o\">=<\/span> <span class=\"n\">re<\/span><span class=\"o\">.<\/span><span class=\"n\">sub<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;\\([^\\)]*\\)&#39;<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;&#39;<\/span><span class=\"p\">,<\/span> <span class=\"n\">summary<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">summary<\/span> <span class=\"o\">=<\/span> <span class=\"n\">re<\/span><span class=\"o\">.<\/span><span class=\"n\">sub<\/span><span class=\"p\">(<\/span><span class=\"s1\">&#39;\\\/[^\\\/]*\\\/&#39;<\/span><span class=\"p\">,<\/span> <span class=\"s1\">&#39;&#39;<\/span><span class=\"p\">,<\/span> <span class=\"n\">summary<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">return<\/span> <span class=\"n\">summary<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">def<\/span> <span class=\"nf\">wolfra<\/span><span class=\"p\">(<\/span><span class=\"n\">question<\/span><span class=\"p\">):<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"s2\">&#34;&#34;<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">client<\/span> <span class=\"o\">=<\/span> <span class=\"n\">wolframalpha<\/span><span class=\"o\">.<\/span><span class=\"n\">Client<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;Your WolframAlpha API Key&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">res<\/span> <span class=\"o\">=<\/span> <span class=\"n\">client<\/span><span class=\"o\">.<\/span><span class=\"n\">query<\/span><span class=\"p\">(<\/span><span class=\"n\">speechtext<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">if<\/span><span class=\"p\">(<\/span><span class=\"nb\">len<\/span><span class=\"p\">(<\/span><span class=\"n\">res<\/span><span class=\"o\">.<\/span><span class=\"n\">pods<\/span><span class=\"p\">)<\/span> <span class=\"o\">&gt;<\/span> <span class=\"mi\">0<\/span><span class=\"p\">):<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">results<\/span> <span class=\"o\">=<\/span> <span class=\"nb\">list<\/span><span class=\"p\">(<\/span><span class=\"n\">res<\/span><span class=\"o\">.<\/span><span class=\"n\">results<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"n\">results<\/span><span class=\"p\">[<\/span><span class=\"mi\">0<\/span><span class=\"p\">]<\/span><span class=\"o\">.<\/span><span class=\"n\">text<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">return<\/span> <span class=\"n\">result<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">r<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sr<\/span><span class=\"o\">.<\/span><span class=\"n\">Recognizer<\/span><span class=\"p\">()<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">m<\/span> <span class=\"o\">=<\/span> <span class=\"n\">sr<\/span><span class=\"o\">.<\/span><span class=\"n\">Microphone<\/span><span class=\"p\">()<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">m<\/span><span class=\"o\">.<\/span><span class=\"n\">RATE<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">44100<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">m<\/span><span class=\"o\">.<\/span><span class=\"n\">CHUNK<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">512<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\">\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">say<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;How can I help you?&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;A moment of silence, please...&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">with<\/span> <span class=\"n\">m<\/span> <span class=\"k\">as<\/span> <span class=\"n\">source<\/span><span class=\"p\">:<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">adjust_for_ambient_noise<\/span><span class=\"p\">(<\/span><span class=\"n\">source<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">if<\/span><span class=\"p\">(<\/span><span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">energy_threshold<\/span> <span class=\"o\">&lt;<\/span> <span class=\"mi\">2000<\/span><span class=\"p\">):<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">energy_threshold<\/span> <span class=\"o\">=<\/span> <span class=\"mi\">2000<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;Set minimum energy threshold to <\/span><span class=\"si\">{}<\/span><span class=\"s2\">&#34;<\/span><span class=\"o\">.<\/span><span class=\"n\">format<\/span><span class=\"p\">(<\/span><span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">energy_threshold<\/span><span class=\"p\">))<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;Say something!&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">audio<\/span> <span class=\"o\">=<\/span> <span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">listen<\/span><span class=\"p\">(<\/span><span class=\"n\">source<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;Got it! Now to recognize it...&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">say<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;one moment, let me think&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">try<\/span><span class=\"p\">:<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">speechtext<\/span> <span class=\"o\">=<\/span> <span class=\"n\">r<\/span><span class=\"o\">.<\/span><span class=\"n\">recognize<\/span><span class=\"p\">(<\/span><span class=\"n\">audio<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;You said: &#34;<\/span> <span class=\"o\">+<\/span> <span class=\"n\">speechtext<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">pattern<\/span> <span class=\"o\">=<\/span> <span class=\"n\">re<\/span><span class=\"o\">.<\/span><span class=\"n\">compile<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;(.*)(tell|say)(.*) about (.*)&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pattern<\/span><span class=\"o\">.<\/span><span class=\"k\">match<\/span><span class=\"p\">(<\/span><span class=\"n\">speechtext<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">if<\/span><span class=\"p\">(<\/span><span class=\"n\">result<\/span> <span class=\"o\">!=<\/span> <span class=\"kc\">None<\/span><span class=\"p\">):<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">answer<\/span> <span class=\"o\">=<\/span> <span class=\"n\">wiki<\/span><span class=\"p\">(<\/span><span class=\"n\">result<\/span><span class=\"o\">.<\/span><span class=\"n\">group<\/span><span class=\"p\">(<\/span><span class=\"mi\">4<\/span><span class=\"p\">))<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">answer<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">say<\/span><span class=\"p\">(<\/span><span class=\"n\">answer<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">answer<\/span> <span class=\"o\">=<\/span> <span class=\"n\">wolfra<\/span><span class=\"p\">(<\/span><span class=\"n\">speechtext<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">if<\/span> <span class=\"ow\">not<\/span> <span class=\"n\">answer<\/span><span class=\"p\">:<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;sorry, I don&#39;t know&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">say<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;sorry, I don&#39;t know&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">else<\/span><span class=\"p\">:<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"n\">answer<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">say<\/span><span class=\"p\">(<\/span><span class=\"n\">answer<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"k\">except<\/span> <span class=\"ne\">LookupError<\/span><span class=\"p\">:<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;sorry, I didn&#39;t catch that&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">say<\/span><span class=\"p\">(<\/span><span class=\"s2\">&#34;sorry, I didn&#39;t catch that&#34;<\/span><span class=\"p\">)<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"><span class=\"k\">while<\/span> <span class=\"kc\">True<\/span><span class=\"p\">:<\/span>\n<\/span><\/span><span class=\"line\"><span class=\"cl\"> <span class=\"n\">listen<\/span><span class=\"p\">()<\/span>\n<\/span><\/span><\/code><\/pre><\/td><\/tr><\/table>\n<\/div>\n<\/div><p>If you want to build a voice assistant, I suggest trying DialogFlow as the response engine. It can make responses more natural, and it can have a conversation to gather all the required information for a task (e.g. book a flight ticket).<\/p>"},{"title":"About Me","link":"https:\/\/chaoming.li\/about-me\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/chaoming.li\/about-me\/","description":"<p>Hello! \ud83d\udc4b<\/p>\n<p>I&rsquo;m Chaoming Li, a tech entrepreneur and software engineer with a passion for building innovative digital solutions that make a real impact.<\/p>\n<p>As the CEO and co-founder of <a class=\"link\" href=\"https:\/\/www.insightech.com\" target=\"_blank\" rel=\"noopener\"\n>Insightech<\/a>, I lead a team dedicated to revolutionizing digital analytics. Our platform helps businesses deeply understand their digital experiences and optimize conversion rates through advanced analytics and insights.<\/p>\n<h2 id=\"technical-expertise\">Technical Expertise\n<\/h2><p>My technical foundation is built on:<\/p>\n<ul>\n<li>Backend development with Go<\/li>\n<li>Frontend development with React<\/li>\n<li>Cloud infrastructure and scalable systems<\/li>\n<li>Digital analytics and optimization<\/li>\n<\/ul>\n<p>I believe in the power of technology to solve real-world problems and am constantly exploring new technologies and methodologies to push the boundaries of what&rsquo;s possible.<\/p>\n<h2 id=\"writing--sharing\">Writing &amp; Sharing\n<\/h2><p>On this blog, I share my experiences and insights about:<\/p>\n<ul>\n<li>Software development best practices<\/li>\n<li>Technology trends and innovations<\/li>\n<li>Entrepreneurship journey<\/li>\n<li>Digital analytics and optimization strategies<\/li>\n<\/ul>\n<h2 id=\"connect-with-me\">Connect With Me\n<\/h2><p>I&rsquo;m always excited to connect with fellow tech enthusiasts, entrepreneurs, and innovators. You can find me on:<\/p>\n<ul>\n<li><a class=\"link\" href=\"https:\/\/www.linkedin.com\/in\/chaomingli\/\" target=\"_blank\" rel=\"noopener\"\n>LinkedIn<\/a> - For professional networking and industry insights<\/li>\n<li><a class=\"link\" href=\"https:\/\/twitter.com\/ChaomingLi\" target=\"_blank\" rel=\"noopener\"\n>Twitter<\/a> - For tech discussions and updates<\/li>\n<li><a class=\"link\" href=\"https:\/\/github.com\/chaoming\" target=\"_blank\" rel=\"noopener\"\n>GitHub<\/a> - For code collaborations and open-source contributions<\/li>\n<\/ul>\n<p>Feel free to reach out if you&rsquo;re interested in digital analytics, tech entrepreneurship, or just want to chat about the latest in technology!<\/p>"},{"title":"Archives","link":"https:\/\/chaoming.li\/archives\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/chaoming.li\/archives\/","description":{}},{"title":"Projects","link":"https:\/\/chaoming.li\/projects\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/chaoming.li\/projects\/","description":"<p>I am passionate about technologies and building things that help people in their work and life. Here are the projects I created and am involved.<\/p>"},{"title":"Search","link":"https:\/\/chaoming.li\/search\/","pubDate":"Mon, 01 Jan 0001 00:00:00 +0000","guid":"https:\/\/chaoming.li\/search\/","description":{}}]}}