[{"content":"Just got a new computer again (new job) and realized I need to reconfigure sudo to work with touch-id. I then realized I didn’t have touch-id enabled for sudo in a long time, the reason is that every OS update it resets the config. Today when enabling it, I have realized that there’s a file called \/etc\/pam.d\/sudo_local.template which does exactly that.\nsudo_local is a filed that will not be overwritten when upgrading between OS versions.\nsudo cp \/etc\/pam.d\/{sudo_local.template,sudo_local} # Edit the file and uncomment this line! auth sufficient pam_tid.so ","permalink":"https:\/\/or-e.net\/posts\/sudo-tid\/","summary":"

Just got a new computer again (new job) and realized I need to reconfigure sudo to work with touch-id.\nI then realized I didn’t have touch-id enabled for sudo in a long time, the reason is that every OS update it resets the config.\nToday when enabling it, I have realized that there’s a file called \/etc\/pam.d\/sudo_local.template<\/code> which does exactly that.<\/p>\n

sudo_local<\/code> is a filed that will not be overwritten when upgrading between OS versions.<\/p>","title":"Support Touch ID for sudo"},{"content":"I had encountred a number of times for the need to give a structured message through gRPC errors The default Go err or status.Errorf functions are simple strings by default.\nfunc (s *Server) LintFile(ctx context.Context, req *LintRequest) (*LintResponse, error) { lintRes, err := lint(ctx, req.GetFile()) if err != nil { \/\/ lintRes is a protbuf message containing detailed error \/\/ for each line in the given file. return nil, status.New(codes.FailedPrecondition, "File isn't valid").WithDetails(lintRes) } return &LintResponse{}, nil } The example above illustrates a situation where you need to know what exactly was wrong with the file, The line number of where the error happened, what’s the reason etc.\n","permalink":"https:\/\/or-e.net\/posts\/grpc-details\/","summary":"

I had encountred a number of times for the need to give a structured message through gRPC errors\nThe default Go<\/code> err or status.Errorf<\/code> functions are simple strings by default.<\/p>\n

func<\/span> (s<\/span> *<\/span>Server<\/span>) LintFile<\/span>(ctx<\/span> context<\/span>.Context<\/span>, req<\/span> *<\/span>LintRequest<\/span>) (*<\/span>LintResponse<\/span>, error<\/span>) {\n<\/span><\/span>\tlintRes<\/span>, err<\/span> :=<\/span> lint<\/span>(ctx<\/span>, req<\/span>.GetFile<\/span>())\n<\/span><\/span>\tif<\/span> err<\/span> !=<\/span> nil<\/span> {\n<\/span><\/span>\t\t\/\/ lintRes is a protbuf message containing detailed error<\/span>\n<\/span><\/span>\t\t\/\/ for each line in the given file.<\/span>\n<\/span><\/span>\t\treturn<\/span> nil<\/span>, status<\/span>.New<\/span>(codes<\/span>.FailedPrecondition<\/span>, "File isn't valid"<\/span>).WithDetails<\/span>(lintRes<\/span>)\n<\/span><\/span>\t}\n<\/span><\/span>\treturn<\/span> &<\/span>LintResponse<\/span>{}, nil<\/span>\n<\/span><\/span>}\n<\/span><\/span><\/code><\/pre><\/div>
The example above illustrates a situation where you need to know what exactly was wrong with the file,\nThe line number of where the error happened, what’s the reason etc.<\/p>","title":"Add Protobuf messages into gRPC errors."},{"content":"Google BigQuery is a very powerful, serverless data warehosue that lets you ingest unlimited data on a pay-per-use basis (storage + querying). The primary advantage of data warehouses is the ability to quickly query and analyze immense amounts of structured data.\nModern data warehouses support new, unstructured data types such as JSON, Avro, and so on, which makes these data warehouses a great contender for data lakes. BigQuery recently added native JSON column type, which we can leverage for our semi-structured lakehouse. With JSON support we can skip the traditional datalakes that are just blob storage with files, from classic Hadoop + hive through GCS, S3, Athena etc. Maintaining such infrastructure and understanding the low-level components, such as metastore, ORC, and Parquet is an expensive process for most startups, financially and with regard to domain expertise.\nData Lakes Let’s take a step back and look at standard data lakes. This will lay the foundation for clearly understanding the benefits of using BigQuery for these modern, advanced workloads that contain unstructred data. Data lakes are built for unstructured data using blob storage systems by using efficient compression and columnar data files, such as Parquet. The files are partitioned using a special directory structure for the given time range, making it easy to scan specific time ranges. Now, ideally we need to make it even faster to query lots of data for a specific event_name. To do this, we need to use another component called metastore.\nMetastore is a meta-data storage system that can index certain properties of the data, which makes it easier to pinpoint the correct files containing the relevant data. We have two separate large-scale systems that data engineers need to manage. Filesystems are cumbersome and much more difficult to both query and maintain.\nIt’s worth noting that blob storage has several clear advantages:\nCheaper to store data. Contains storage tiers, such as archival, cold, and warm. Can hold data of any type or size (audio, images, executables, etc.). Real unstructured data like audio can be used for ML piplines, text-to-speech algorithms, etc. Easy to replicate across regions and providers. What’s a data lakehouse? A data warehouse is storage for structured data. A data lake is storage for unstructured data. A data lakehouse is the beautiful storage love child of the warehouse and lake. The lakehouse leverages the semi-structured JSON column of BigQuery and helps us ingest a flexible data structure while the native columns act as our metastore.\nLakehouse example In this example, we use several columns for partioning (timestamp) and clustering (event_name, source). They act as metadata to our semi-structured data. This way it’s easy to navigate and explore data via SQL and to narrow down the subset of data we need. We can then use the new native JSON notation in our query:\nSELECT id, timestamp, event_name, data.hello FROM datalake Next, we use materialized-views or scheduled queries (via BigQuery\/Airflow etc.) to build our final facts and dimensions tables.\nid timestamp event_name source content_type data data_bytes 6EF89A2E-8E11-4848-91B3-11682020559F Sun Mar 6 15:48:24 IST 2022 something_happened service-a application\/json {"hello": "from lakehouse"} NULL Easy ETLs with materialized-views IMHO SQL is a great language to use to describe data. SQL is making a noticeable come back. Even Elastic and MongoDB support SQL. The rise of new-SQL databases like CockroachDB, modern warehouses, logging systems, etc. are clear evidence that SQL is here to stay.\nWe will use a cheap form of ETL using BigQuery’s materialized-view feature. The default refresh rate for materialized-view is 30m, you can of course tweak that, but it’s sufficient for most workloads.\nIf you already have such system in place you can use it as well. See: Airflow, dbt or just a simple Kubernetes job.\nCREATE MATERIALIZED VIEW `<project_id>.<dataset>.canary_tests` PARTITION BY TIMESTAMP_TRUNC(timestamp, HOUR) CLUSTER BY service_name AS SELECT timestamp, data.service_name AS service_name, data.job_name AS job_name, data.workflow_id AS workflow_id, data.test_name AS test_name, data.build_url as build_url, FROM `<project_id>.<dataset>.datalake_v1` WHERE event_name = "something_happened" Conclusion BigQuery is an excellent candidate to serve as the basis for your next data lake adventure (good luck!).\nBigQuery can help you start small and easily scale out. Organizations of all sizes will benefit (if you don’t store anything you won’t pay for it). Data engineering teams can build an impressive data platform levereging the power of SQL. ","permalink":"https:\/\/or-e.net\/posts\/bigquery-lakehouse\/","summary":"
Google BigQuery is a very powerful, serverless data warehosue that\nlets you ingest unlimited data on a pay-per-use basis (storage + querying).\nThe primary advantage of data warehouses is the ability to quickly query and analyze immense amounts of structured<\/strong> data.<\/p>\n
Modern data warehouses support new, unstructured data types such as JSON, Avro, and so on, which makes these data warehouses a great contender for data lakes.\nBigQuery recently added native JSON column type, which we can leverage for our semi-structured lakehouse.\nWith JSON support we can skip the traditional datalakes that are just blob storage with files, from classic Hadoop + hive through GCS, S3, Athena etc.\nMaintaining such infrastructure and understanding the low-level components, such as metastore, ORC, and Parquet is an expensive process for most startups, financially and with regard to domain expertise.<\/p>","title":"Build a datalake on top of BigQuery"},{"content":"Or is an experienced SRE for over a decade, building companies and their infrastructure. Responsible for educating engineers working with cloud-native platforms on best-practices, helping them be the best and most efficient they can be in order to provide value with least friction. Designing highly scalable platforms that serve billions of requests and happy customers with resiliency and simplicity in mind.\nManifesting systems and bringing them to life with the best in class technology-stack that fits the needs of the business. Whether it be Kubernetes, Google Cloud Platform, AWS, Go, Prometheus, Temporal etc. Writing packages alongside developers to help them abstract the complexity away from servers they build such as gRPC and Pub\/Sub with developer-experience in mind.\nLeading the compliance processes (SOC2, ISO, PCI, HIPAA etc.) with implementing security guidelines and guardrails altogether with education and presentations preventing bad practices to reoccur in the future.\nStriving for data-driven organizations Or also designed the BI and data engineering practices, data-pipelines, dashboards and tools helping organizations emitting data to lakes and warehouses by providing libraries and SDKs to interact with massive data needs.\nPipelines written in Apache-beam, dashboards with Apache Superset or Metabase, headless BI for frontends and other needs implemented with cube.dev\nCreating and advocating for advanced methodologies together with developers to have common ground between SRE and devs to have great DevOps culture and to have them impact their daily work.\nCI\/CD with progressive rollouts and canaries Feature flags and A\/B testing GitOps Ownership of teams and on-call shifts (you build it, you run it) Postmortem framework with emphasis on being blameless in order to learn and set up prevention for the future ChatOps and chatbots that help devs debug and understand the system without the need of installing tools\/scripts by providing a lot of the tasks as a Slack bot to interact with many daily tedious tasks that are automated via simple commands Contributing to the open-source community for tools we use and love. Publishing internal tools as open-source when it’s clear it has more general use cases outside the company and can make the community’s lives easier.\nSRE lead (Founding team) - Mind (2023 - 2025) Built CI pipelines using Dagger, contributed to the core product reliability and scalability. Leveraging Google Cloud Platform, GKE. Managed inference infrastructure using various models.\nSRE lead (Founding team) \u2014 Torq (2020 - 2023) As a founding-team member and SRE lead at torq Or designed and built the cloud-infrastructure. Wrote crticial product components. Leading many architectural decisions and implementations.\nSRE lead \u2014 Apester (2017 - 2020) Migrated the company from AWS EC2 to GCP and Kubernetes, building the new data infrastructure on top of BigQuery, Pub\/Sub and Dataflow.\nDevOps Engineer \u2014 AppCoin (2015 - 2017) Designed and built from scratch an entire infrastructure on Google Cloud Platform on top of Kubernetes and Docker containers. Continuous deployment, Monitoring (Prometheus & Grafana). Security policies and threat management.\nDevOps Engineer \u2014 Wix.com (2014 - 2015) Infrastructure solutions, working on Google Cloud Platform, AWS and physical infra. Mail anti-spoofing implementation using DKIM, DMARC etc. Monitoring infra at large scale with Sensu and InfluxDB. Design new infrastructure with enhanced security, firewalls etc.\nDevOps Engineer \u2014 XWise a Playtech company (2011 - 2014) Deployed and managed a highly available e-mail Infrastructure. Maintained and designed the firewalls and other security aspects including VPNs, Security audits regulations and certifications, disaster recovery policies and access control of the Organisation. Administer private and public clouds on vSphere and Amazon respectively, Providing all the related systems: fibre switches, SANs, vSwitches and Security-wise actions. Designed the company\u2019s monitoring and centralised log systems (Nagios, ELK etc.). Extremely experienced with configuration management systems (Puppet). Version control systems (Git, Subversion). Database administration and development on Relational and NoSQL databases (PostgreSQL, Redis, MongoDB). Writing programs, scripts and web tools. Maintained a large-scale and distributed web environments.\nExternal links Linkerd case study: https:\/\/www.cncf.io\/case-studies\/apester\/ Google Cloud case study: https:\/\/cloud.google.com\/customers\/apester ","permalink":"https:\/\/or-e.net\/about\/","summary":"
Or is an experienced SRE for over a decade, building companies and their infrastructure.\nResponsible for educating engineers working with cloud-native platforms on best-practices,\nhelping them be the best and most efficient they can be in order to provide value with least friction.\nDesigning highly scalable platforms that serve billions of requests and happy customers with resiliency and simplicity in mind.<\/p>\n
Manifesting systems and bringing them to life with the best in class technology-stack that fits the needs of the business.\nWhether it be Kubernetes, Google Cloud Platform, AWS, Go, Prometheus, Temporal etc.\nWriting packages alongside developers to help them abstract the complexity away from servers they build\nsuch as gRPC and Pub\/Sub with developer-experience in mind.<\/p>","title":"About"}]