[{"content":"For the past two years, I’ve been part of the SDLC Security team at Datadog. Our core mission is to secure the CI\/CD by providing guidance and tools for other teams to use. If you wanna learn more about what it means in practice, I co-wrote a couple of blog posts about two projects I worked on:\nhttps:\/\/www.datadoghq.com\/blog\/container-image-signing\/ https:\/\/www.datadoghq.com\/blog\/engineering\/malicious-pull-requests\/ I also presented this latest project at the Datadog London Summit in 2026:\nhttps:\/\/youtu.be\/adnGvTTL5vA See you next time \ud83d\udc4b\n","date":"2025-11-16","permalink":"https:\/\/juliendoutre.github.io\/posts\/datadog\/","section":"Posts","summary":"What I do at work","title":"A couple of blog posts about projects I worked on at Datadog"},{"content":"","date":null,"permalink":"https:\/\/juliendoutre.github.io\/","section":"Julien Doutre","summary":"","title":"Julien Doutre"},{"content":"","date":null,"permalink":"https:\/\/juliendoutre.github.io\/posts\/","section":"Posts","summary":"","title":"Posts"},{"content":"GitHub allows to define template repositories which content can be used as a base for new ones.\nI created such a template myself at https:\/\/github.com\/juliendoutre\/template. It contains a CODEOWNER, a .gitignore, a README.md, a LICENSE, and a dependabot.yaml files configured for ecosystems I use often.\nUnfortunately, templating does not apply settings to a repository. In order to do this, I created https:\/\/github.com\/juliendoutre\/factory, a script that enables me to enforce settings on existing repositories, or create new ones with safe defaults.\nThe heavy lifting of the tool is done with Terraform which I like the declarative syntax and readability very much. I defined a few resources using the official GitHub Terraform provider.\nI use the lifecycle { ignore_changes = [...] } block to ignore some values that could be problematic. For instance, as I set the template { ... } block to point to my template repository, new repos created by the script are based on it, but older ones always show a diff in the Terraform plan. The tool is then simply a bash script that dynamically import resources if they already exist into the Terraform state. It ultimately runs an apply that prints out the planned changes and asks the user if they wanna go ahead with them.\nI’m kind of abusing the Terraform state management, as I’m using a local state file that I discard as soon as the script returns. But, this let’s me see clearly what changes will be made by the script and without having to deal with any complex programmatic logic.\nFeel free to try it based on other templates or with different settings and let me know what you think!\nSee you next time \ud83d\udc4b\n","date":"2025-01-23","permalink":"https:\/\/juliendoutre.github.io\/posts\/factory\/","section":"Posts","summary":"How I manage my personal projects GitHub configuration","title":"Enforcing GitHub repository settings with Terraform"},{"content":"In 2022, I ran Vincennes’ half marathon with a friend.\nAfter the race, I was curious to know how I did compared to other participants. Fortunately, the results were made available online at https:\/\/protiming.fr\/Results\/runningR\/6294\/14\/.\nI figured out a way of paginating through the results after having clicked a few buttons. Adding page:N at the end of the URL seemed to be good enough.\nWhen the page number exceedes the last one, the website returns a 404 HTTP status code so it’s easy to know when to stop iterating.\nAfter spending some time inspecting the page source code, I came up with the following Python script:\nimport requests import json from bs4 import BeautifulSoup URL = "https:\/\/protiming.fr\/Results\/runningR\/6294\/14\/page:{}" DATASET = "data.json" def get_dataset(): times = [] for page in range(1, 156): response = requests.get(URL.format(page)) soup = BeautifulSoup(response.text, features="html.parser") for row in soup.find(id="results").find("tbody").find_all("tr"): times.append({ "time": sum(x * int(t) for x, t in zip([3600, 60, 1], row.find("td", {"class": "real_time_data"}).text.split(":"))), "category": row.find_all("td")[5].text.strip().lower(), }) return times def main(): data = get_dataset() with open(DATASET, "w") as f: json.dump(data, f) if __name__ == "__main__": main() I used the famous BeautifulSoup Python library to extract HTML tags of interest and saved the whole results as a JSON with the following format:\n[ { "time": 4209, \/\/ in seconds "category": "m0m (1 \/ 454)" }, [...] ] I only cared about people’s time and their category. This latest field is a string formatted in a way you can extract the sex and age of the participant. For instance, when the second character is a m, the record is for a male, and similarly, f indicates a female.\nOnce I had the dataset locally (I did not want to DOS the website by scrapping it too often), I could start playing with it.\nI was mostly interested in drawing the general histogram of times, and then one for each sex. I ended up with the following code:\nimport json import matplotlib.pyplot as plt DATASET = "data.json" def load_dataset(): with open(DATASET, "r") as f: return json.load(f) def group_by_sex(data): women = [] men = [] for datum in data: if datum["category"][2] == "f": women.append(datum) elif datum["category"][2] == "m": men.append(datum) return women, men def display_histogram(data, bins=200): plt.hist([point["time"] for point in data], bins=bins) plt.show() def main(): data = load_dataset() display_histogram(data) groups = group_by_sex(data) for group in groups: display_histogram(group) if __name__ == "__main__": main() This allowed to generate the following histograms: Distribution of all participants times Distribution of female participants times Distribution of male participants times With my time (1:46:56, therefore 6416 seconds) it ended up I’m pretty average \u02c6\u02c6\nSee you next time \ud83d\udc4b\n","date":"2025-01-22","permalink":"https:\/\/juliendoutre.github.io\/posts\/half-marathon-vincennes\/","section":"Posts","summary":"Web scraping and histograms","title":"Some statistics about Vincennes' half marathon (2022)"},{"content":"Let’s say you have a process that performs many actions involving dependency resolution and fetching, requesting various APIs, sending telemetry, etc. Something like a CI runner compiling a program for instance. How can you make sure to catch all artifacts it pulls from the outside (namely the Internet) or data it sends out?\nI tried using eBPF and Frida to achieve this but both proved to be difficult to use:\neBPF can intercept TCP packets in kernel space but TLS connections are negotiated by programs in user space eBPF (or Frida) can attach probes to functions in user space, but depending on the languages at stake, different libs may be used to perform HTTP requests. Supporting all of them is too dauting of a task. A simpler approach would be to have HTTP requests go through a proxy server. The HTTP_PROXY and HTTPS_PROXY environment variables are a de facto standard that many tools abide by. Setting them will make sure that requests are proxied to the URL their values hold instead of being sent directly to the target.\nSetting a couple of environment variables is a good enough trade off on my side if it allows an otherwise transparent setup for the user. My end goal was to build a tool that one could wrap commands with, to monitor all their HTTP requests and get a report of what APIs were reached and what dependencies where pulled. So let’s get started!\nmkdir proxaudit cd .\/proxaudit go mod init github.com\/juliendoutre\/proxaudit git init You can find the project source code on https:\/\/github.com\/juliendoutre\/proxaudit. Check out the commits history to see each iteration! First we need to create a HTTP server that will listen on a given port.\nserver := &http.Server{ Addr: ":" + strconv.FormatUint(*port, 10), Handler: &handler{logger}, ReadTimeout: 10 * time.Second, ReadHeaderTimeout: 5 * time.Second, IdleTimeout: 120 * time.Second, } Here logger is a *zap.Logger and port is an uint64 value provided by the user through a command line flag. I set up some default timeouts as a good security practice but the values are not critical to the implementation.\nhandler is a custom struct that implements the http.Handler interface defined as:\ntype Handler interface { ServeHTTP(ResponseWriter, *Request) } I wrote it as:\ntype handler struct { logger *zap.Logger } func (h *handler) ServeHTTP(rw http.ResponseWriter, req *http.Request) { h.logger.Info("received request", zap.String("method", req.Method), zap.String("url", req.URL.String())) httputil.NewSingleHostReverseProxy(req.URL).ServeHTTP(rw, req) } The ServeHTTP method is not doing much. It logs the request’s HTTP verb and the destination URL before deferring the proxy work to a httputil.ReverseProxy struct available in Go’s standard library.\nIf I put everything together in a main.go file:\nfunc main() { port := flag.Uint64("port", 8000, "port to listen on") flag.Parse() logger, err := zap.NewProductionConfig().Build() if err != nil { log.Panic(err) } server := &http.Server{ Addr: ":" + strconv.FormatUint(*port, 10), Handler: &handler{logger}, ReadTimeout: 10 * time.Second, ReadHeaderTimeout: 5 * time.Second, IdleTimeout: 120 * time.Second, } go waitForSignal(server, logger) logger.Info("Starting HTTP proxy server...", zap.Uint64("port", *port)) if err := server.ListenAndServe(); err != nil { if errors.Is(err, http.ErrServerClosed) { logger.Warn("HTTP proxy server was stopped", zap.Error(err)) } else { logger.Panic("Failed running HTTP proxy server", zap.Error(err)) } } } and start the server with go run .\/main.go it should log something like:\n{"level":"info","ts":1733854054.4425678,"caller":"proxaudit\/main.go:38","msg":"Starting HTTP proxy server...","port":8000} and then hang, waiting for requests.\nwaitForSignal is a util function I wrote to catch signals such as SIGKILL and gracefully terminate the server:\nfunc waitForSignal(server *http.Server, logger *zap.Logger) { stop := make(chan os.Signal, 2) signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM) <-stop if err := server.Shutdown(context.Background()); err != nil { logger.Error("Failed shutting down HTTPS proxy server", zap.Error(err)) } } If I open a shell in another tab and run http_proxy=http:\/\/localhost:8000 curl http:\/\/google.com I can see the proxy logging some requests:\n{"level":"info","ts":1733854063.6379368,"caller":"proxaudit\/main.go:64","msg":"received request","method":"GET","url":"http:\/\/google.com\/"} We got a plain HTTP request proxy working!\nCode checkpoint Nowadays though, most people use HTTPS. Running http_proxy=http:\/\/localhost:8000 curl https:\/\/google.com does not result in any log on the proxy side. We could have expected this, we changed of protocol. But what happens if I run HTTPS_PROXY=http:\/\/localhost:8000 curl https:\/\/google.com?\nThe server logs an error: http: proxy error: unsupported protocol scheme "" and curl returns an error as well: curl: (56) CONNECT tunnel failed, response 502.\nWhy is this? I understood the issue while reading https:\/\/eli.thegreenplace.net\/2022\/go-and-proxy-servers-part-2-https-proxies\/.\nA HTTPS connection can go through proxies but will only send a CONNECT request to them, indicating which host it targets. It’s expecting the underlying TLS connection to be tunneled as is.\nIn order to fix my code, I extended my ServeHTTP function like this:\nfunc (h *handler) ServeHTTP(rw http.ResponseWriter, req *http.Request) { h.logger.Info("received request", zap.String("method", req.Method), zap.String("url", req.URL.String())) if req.Method == http.MethodConnect { h.handleConnect(rw, req) } else { httputil.NewSingleHostReverseProxy(req.URL).ServeHTTP(rw, req) } } func (h *handler) handleConnect(rw http.ResponseWriter, req *http.Request) { hijacker, ok := rw.(http.Hijacker) if !ok { http.Error(rw, "Hijacking not supported", http.StatusInternalServerError) return } clientConn, _, err := hijacker.Hijack() if err != nil { http.Error(rw, err.Error(), http.StatusServiceUnavailable) return } defer clientConn.Close() serverConn, err := net.Dial("tcp", req.Host) if err != nil { http.Error(rw, err.Error(), http.StatusServiceUnavailable) return } defer serverConn.Close() clientConn.Write([]byte("HTTP\/1.1 200 Connection Established\\r\\n\\r\\n")) go io.Copy(serverConn, clientConn) io.Copy(clientConn, serverConn) } The handleConnect method really just returns a success status to the client and then recover the underlying TCP connection (thanks to the http.Hijacker interface casting) to simply forward all the traffic to the destination host.\nNow I can run HTTPS_PROXY=http:\/\/localhost:8000 curl https:\/\/google.com without errors!\nCode checkpoint But… it’s simply logging CONNECT requests, not the underlying HTTPS requests. As the proxy blindly forward the TLS connection to the host, it does not have access to the raw content. And it’s why TLS is used for, right? Preventing eavesdroppers from intercepting cleartext traffic. Fortunately, there’s a way to overcome this though by using Man-In-The-Middle (MITM) certificate crafting.\nI read some documentation from the mitmproxy Python project to understand the technique at play. The idea is to craft a fake certificate using a Certificate Authority (CA) under our control and that is trusted by the client to terminate the connection in the proxy. Then open a second connection to the server to forward requests after they’ve been handled by the proxy. This requires to fetch the destination’s certificate to craft a matching certificate.\nAs my end goal is to make an open source observability tool, I can ask users to trust a certificate that would intercept their traffic. I first ran:\nopenssl genrsa -out ca.key 4096 openssl req -new -x509 -days 365 -key ca.key -out ca.crt openssl req -newkey rsa:4096 -nodes -keyout server.key -subj "\/CN=localhost" -out server.csr openssl x509 -req -extfile <(printf "subjectAltName=DNS:localhost") -days 365 -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt to generate a CA and a certificate for my proxy before a colleague told me about https:\/\/github.com\/FiloSottile\/mkcert:\nbrew install mkcert mkcert -install mkcert localhost I used the excellent https:\/\/github.com\/elazarl\/goproxy Go module to perform the MITM interception for me:\nfunc main() { \/\/ [...] proxy := goproxy.NewProxyHttpServer() proxy.Verbose = false proxy.OnRequest().HandleConnect(goproxy.AlwaysMitm) proxy.OnRequest(goproxy.ReqConditionFunc( func(req *http.Request, _ *goproxy.ProxyCtx) bool { return req.Method != http.MethodConnect }, )).DoFunc( func(req *http.Request, _ *goproxy.ProxyCtx) (*http.Request, *http.Response) { logger.Info("received a request", zap.String("method", req.Method), zap.String("url", req.URL.String())) return req, nil }, ) \/\/ [...] } I configured the proxy to mitm connect requests and log all other methods. I replaced my custom handler with it:\nserver := &http.Server{ Addr: ":" + strconv.FormatUint(*port, 10), Handler: proxy, \/\/ this line changed ReadTimeout: 10 * time.Second, ReadHeaderTimeout: 5 * time.Second, IdleTimeout: 120 * time.Second, } and loaded the CA I created with mkcert before starting it:\nfunc main() { mkcertDir := path.Join(os.Getenv("HOME"), "Library", "Application Support", "mkcert") port := flag.Uint64("port", 8000, "port to listen on") caCertPath := flag.String("ca-cert", path.Join(mkcertDir, "rootCA.pem"), "path to a CA certificate") caKeyPath := flag.String("ca-key", path.Join(mkcertDir, "rootCA-key.pem"), "path to a CA private key") flag.Parse() logger, err := zap.NewProductionConfig().Build() if err != nil { log.Panic(err) } cert, err := tls.LoadX509KeyPair(*caCertPath, *caKeyPath) if err != nil { logger.Panic("Failed loading CA certificate", zap.Error(err)) } goproxy.GoproxyCa = cert \/\/ [...] } and now HTTPS interception works like a charm!\n{"level":"info","ts":1733868127.719048,"caller":"proxaudit\/main.go:66","msg":"Starting HTTP proxy server...","port":8000} {"level":"info","ts":1733868147.167201,"caller":"proxaudit\/main.go:50","msg":"received a request","method":"GET","url":"http:\/\/google.com\/"} {"level":"info","ts":1733868150.407752,"caller":"proxaudit\/main.go:50","msg":"received a request","method":"GET","url":"http:\/\/google.com\/"} {"level":"info","ts":1733868150.427733,"caller":"proxaudit\/main.go:50","msg":"received a request","method":"GET","url":"http:\/\/www.google.com\/"} {"level":"info","ts":1733868156.662025,"caller":"proxaudit\/main.go:50","msg":"received a request","method":"GET","url":"https:\/\/google.com:443\/"} {"level":"info","ts":1733868160.218765,"caller":"proxaudit\/main.go:50","msg":"received a request","method":"GET","url":"https:\/\/google.com:443\/"} {"level":"info","ts":1733868160.5086799,"caller":"proxaudit\/main.go:50","msg":"received a request","method":"GET","url":"https:\/\/www.google.com:443\/"} ^C{"level":"warn","ts":1733868173.727085,"caller":"proxaudit\/main.go:70","msg":"HTTP proxy server was stopped","error":"http: Server closed"} Code checkpoint Some tools may require more than setting HTTPS_PROXY to work. For instance:\nHTTPS_PROXY=https:\/\/localhost:8000 npm i dd-trace alone does not work. One need to run:\nNODE_EXTRA_CA_CERTS="$(mkcert -CAROOT)\/rootCA.pem" HTTPS_PROXY=https:\/\/localhost:8000 npm i dd-trace to allow our local CA to emit trusted certificates.\nNote that for old versions of NPM, there was an issue documented at https:\/\/github.com\/dependabot\/dependabot-core\/issues\/10623 when using goproxy. Introducing proxaudit! #The final stage for me was to use all this logic in a tool that can wrap any command and observe its HTTP(s) requests thanks to a proxy. Check out https:\/\/github.com\/juliendoutre\/proxaudit to see the project’s latest version!\nbrew tap juliendoutre\/proxaudit https:\/\/github.com\/juliendoutre\/proxaudit brew install proxaudit mkcert -install proxaudit -- curl http:\/\/google.com proxaudit -- curl https:\/\/google.com proxaudit # Read from stdin proxaudit -output logs.jsonl -- curl https:\/\/google.com # Write logs to file I took inspiration from https:\/\/github.com\/99designs\/aws-vault for the design (especially the way they forward signals to subprocesses).\nLet me know if anything is missing or if you found a bug by opening an issue \ud83d\ude4f\nSee you next time \ud83d\udc4b\n","date":"2024-12-09","permalink":"https:\/\/juliendoutre.github.io\/posts\/proxaudit\/","section":"Posts","summary":"An experiment and side project about HTTPS proxying in Go","title":"Writing a HTTPS proxy in Golang"},{"content":"Last year a coworker shared with me the excellent https:\/\/gtfobins.github.io\/ website, a compendium of techniques to abuse misconfigurations in Linux binaries. It’s an open source project and its code can be found on GitHub at https:\/\/github.com\/GTFOBins\/GTFOBins.github.io.\nI wanted to build a discovery tool leveraging this data set. I had some time on a week-end and created https:\/\/github.com\/juliendoutre\/gogtfobins\/ to do so. It’s a Go CLI built with the cobra library exposing three commands:\ngogtfobins list to list all binaries available on the host and the functions they can eventually allow to obtain gogtfobins describe BINARY to print some details about a specific binary gogtfobins exploit BINARY FUNCTION to run an exploit for a binary Here is a more concrete example:\n# List all available binaries allowing for opening a reverse shell on the current host. gogtfobins list --function reverse-shell # Print possible exploits for the docker binary. gogtfobins describe docker # Get a reverse-shell using the docker binary. gogtfobins exploit docker reverse-shell You can install it easily with homebrew:\nbrew tap juliendoutre\/gogtfobins https:\/\/github.com\/juliendoutre\/gogtfobins brew install gogtfobins or download a built binary from the available releases.\nThe gtfobins data is embedded thanks to a go embed directive that is then used to build both an index and a reverse index. Commands simply query this index. You can actually reuse this data structure in your own project as it is exposed in a Go module:\ngo get github.com\/juliendoutre\/gogtfobins Let me know if anything is missing or you found a bug by opening an issue.\nSee you next time \ud83d\udc4b\n","date":"2024-12-03","permalink":"https:\/\/juliendoutre.github.io\/posts\/go-gtfobins\/","section":"Posts","summary":"I developed a Go program to exploit misconfigurations in Linux binaries based on https:\/\/gtfobins.github.io<\/a>","title":"Living off the land in Linux!"},{"content":"I always wanted to create a video game. It’s a topic I approached a bit while working on some projects like a solution to the Synacor challenge or a R8 emulator implementation. But there were mostly coding challenges and did not involve any actual game design.\nI played a bit with Unity and even created a small platformer game with some friends for a school project (play it online here!). However I was confused by the code organization. I really wanted to manage the whole app as code and could not really achieve this. Everything had to be managed through scripts attached to objects only tracked in the IDE or weird metadata files. This was not satisfying to my nascent software engineering mindset!\nA couple of years ago, I stumbled upon the Amethyst project. I was really excited about the ECS approach and read some guides but never came to create anything concrete. And then the project stopped \ud83d\ude22\nHowever some developpers seemed decided to continue the adventure and started Bevy. It’s really close to Amethyst concept-wise but they seemed to get rid of some of the cumbersome type declarations and only kept the best from this framework.\nFor those not familiar with ECS, it stands for Entity-Component-Systems. It’s a way to design games based on entities which can get attached components and updated by systems which updates their state.\nI decided to give it a try and started a new GitHub repository: https:\/\/github.com\/juliendoutre\/froggy.\nI took inspiration from https:\/\/github.com\/bevyengine\/bevy\/blob\/latest\/examples\/games\/game_menu.rs to create a simple splash screen and a game menu with two buttons.\nAssets come from https:\/\/www.kenney.nl which provides an amazing collection of game assets for free \u2764\ufe0f\nDesigning UI components on a canvas was similar as writing HTML nodes but in Rust… which felt rather cumbersome. Once I had a satisfying rendering, I decided to release my game. I skimmed through https:\/\/bevy-cheatbook.github.io\/platforms.html and noticed Bevy support WASM!\nAs for the previous game I was working for, I decided to release it on GitHub pages (free hosting for the win) but this time I wanted to automate this a bit. And it happens the Rust toolchain is pretty well integrated in the GitHub actions ecosystem.\nMy first step was to add a CI workflow with several jobs to check code’s formating, run clippy, tests, build the project, and check the lock file is up to date.\nname: CI # The workflow should only run for commits in PRs and the main branch. on: push: branches: - main pull_request: branches: - '*' # Let's use concurrency groups to cancel stale jobs except on the main branch. concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: ${{ github.ref != 'main' }} # Always explicitly set a workflow permissions! permissions: contents: read One optimization I used is to cache the .target and .cargo folders so that they can be used across jobs. All my jobs therefore start with the following steps:\njobs: my-job: runs-on: ubuntu-latest steps: - uses: actions\/checkout@v4 - uses: actions\/cache@v4 with: path: | ~\/.cargo\/bin\/ ~\/.cargo\/registry\/index\/ ~\/.cargo\/registry\/cache\/ ~\/.cargo\/git\/db\/ target\/ key: ${{ runner.os }}-cargo-${{ hashFiles('**\/Cargo.lock') }} - run: rustup update stable && rustup default stable I noticed compilation errors at build time because of missing dev libraries that I was able to fix simply with:\n- run: sudo apt-get update && sudo apt-get install -y libasound2-dev libudev-dev Then I created a CD workflow to build and deploy the game to a GitHub page:\nname: CD # The workflow should only run for commits on the main branch. on: push: branches: - main # Always explicitly set a workflow permissions! permissions: contents: read with the following jobs:\njobs: build: runs-on: ubuntu-latest steps: - uses: actions\/checkout@v4 - uses: actions\/cache@v4 with: path: | ~\/.cargo\/bin\/ ~\/.cargo\/registry\/index\/ ~\/.cargo\/registry\/cache\/ ~\/.cargo\/git\/db\/ target\/ key: ${{ runner.os }}-cargo-${{ hashFiles('**\/Cargo.lock') }} - run: rustup update stable && rustup default stable # We need to make sure the Rust toolchain supports WASM. - run: rustup target install wasm32-unknown-unknown # Building a WASM binary. - run: cargo build --release --target wasm32-unknown-unknown # Installing some tools to optimizie the WASM binary. - run: cargo install wasm-bindgen-cli@0.2.92 wasm-opt@0.116.1 # Generating some JS code to load the WASM in a HTML canvas. - run: wasm-bindgen --no-typescript --target web --out-dir .\/build\/ --out-name froggy .\/target\/wasm32-unknown-unknown\/release\/froggy.wasm # Optimizing the binary for size. Experimentally, it decreased the size by 2 which saves some bandwidth for the website users (from 30M to 15M). - run: wasm-opt .\/build\/froggy_bg.wasm -o .\/build\/froggy_bg.wasm -Oz # Adding a dead simple HTML file to load the JS code and define the aforementioned canvas. - run: cp .\/www\/index.html .\/build\/index.html # Copying the assets into the build folder so that they are bundled too and served by the website. - run: cp -r .\/assets .\/build\/assets # Uploading the build folder to artifacts. - uses: actions\/configure-pages@v5 - uses: actions\/upload-pages-artifact@v3 with: path: .\/build deploy: # GitHub pages are now action based and not simply based on a Git branch. permissions: pages: write id-token: write environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} runs-on: ubuntu-latest needs: build steps: - uses: actions\/deploy-pages@v4 id: deployment It takes about 9 minutes to build and deploy the website. And here is the final result: https:\/\/juliendoutre.github.io\/froggy!\nThis was a nice journey but I noticed some caveats:\nbevy does not support hot reloading writing Rust code does not let any room for quick hacks which is often needed when developing a game. In the end, most of the points mentioned in https:\/\/loglog.games\/blog\/leaving-rust-gamedev.\nSo next, I’d like to try Godot and give another chance to more “classic” game engines.\nSee you next time \ud83d\udc4b\n","date":"2024-05-11","permalink":"https:\/\/juliendoutre.github.io\/posts\/bevy\/","section":"Posts","summary":"But in the end I just messed around with GitHub actions","title":"I (almost) wrote a game in Rust with Bevy!"},{"content":" Walkthrough of an encoding\/decoding library implementation. Introduction #It’s been a long time I’ve been fascinated by lexers, parsers and more broadly programming language theory. Since data is just a bunch of bits, how do computer split them into recognizable types? And conversely, how can they serialize in-memory types to chunks of data exchangeable over the wire?\nSince I better understand when I actually implement things, I decided to create my own parsing library and share the journey on this website. I chose to write it in Golang to avoid the burden of thinking too much about memory management, at first. Instead, I focused on API design and writing concise code (I hope).\nThis post is in part inspired by the excellent https:\/\/github.com\/rust-bakery\/nom library which I recomment checking out! It’s written in Rust \ud83e\udd80, and relies on a very modular approach to parsing which I really appreciated reading through.\nLet’s get started!\nInitialization #First of all, we need to create a new Go project:\ncd ~\/dev mkdir -p godec # I'm bad at naming cd godec go mod init github.com\/juliendoutre\/godec git init code . Basic interface design #A data parsing library must provide two features:\ndecoding a blob of data to interpret it as some in-memory value (aka unmarshalling, deserializing) encoding some in-memory value into a blob of data (aka marshalling, serializing). We can start by creating a new Go file godec.go defining two very basic interfaces:\n\/\/ godec.go package godec type Encoder interface { Encode() ([]byte, error) } type Decoder interface { Decode(input []byte) ([]byte, error) } Note the Decode method also returns a slice of bytes. It’s the remainder of the input once the Decoder has been applied to it. This way we can easily chain Decoders together. This will be useful later.\nWe call a Codec any object that implements both:\n\/\/ godec.go [...] type Codec interface { Encoder Decoder } But enough abstractions for now, let’s get to some concrete applications!\nA simple use case: color codes #Let’s start with something simple:\n#3399ff <-- I'm blue, da ba dee... A color code starts with a # followed by three two-digits hexadecimal numbers, matching the red, green and blue weights of the encoded color.\nWe can see an appropriate Decoder as a state machine:\nflowchart LR; data([\"`data: []byte`\"]); red([\"`red: uint8`\"]); green([\"`green: uint8`\"]); blue([\"`blue: uint8`\"]); style data fill:#FFB7C5; style red fill:#66CDAA; style green fill:#66CDAA; style blue fill:#66CDAA; hashtag[\"#\"]; hex1[\"two-digits hex number\"]; hex2[\"two-digits hex number\"]; hex3[\"two-digits hex number\"]; termination[\"no more bytes\"]; data --> hashtag; hashtag --> hex1; hex1 --> hex2; hex2 --> hex3; hex3 --> termination; hex1 -.-> red; hex2 -.-> green; hex3 -.-> blue; And an Encoder as its “opposite”: flowchart LR; data([\"`data: []byte`\"]); red([\"`red: uint8`\"]); green([\"`green: uint8`\"]); blue([\"`blue: uint8`\"]); style data fill:#66CDAA; style red fill:#FFB7C5; style green fill:#FFB7C5; style blue fill:#FFB7C5; hashtag[\"#\"]; hex1[\"two-digits hex number\"]; hex2[\"two-digits hex number\"]; hex3[\"two-digits hex number\"]; termination[\"no more bytes\"]; hashtag --> hex1; hex1 --> hex2; hex2 --> hex3; red -.-> hex1; green -.-> hex2; blue -.-> hex3; hex3 --> termination; termination --> data; We need to build a few components already:\na box matching a given bytes slice (here simply #) a box tying an uint8 variable to a two-digits hexadecimal number a box checking there’re no bytes left a way to chain the boxes together Let’s write our first parser’s code:\n\/\/ codecs.go package godec import ( "bytes" "fmt" ) type ExactMatch []byte func (e ExactMatch) Encode(input []byte) ([]byte, error) { return e, nil } func (e ExactMatch) Decode(input []byte) ([]byte, error) { if bytes.Equal(e, input[:len(e)]) { return input[len(e):], nil } return nil, fmt.Errorf("expected %q", string(e)) } var _ Codec = ExactMatch{} I’m using type aliasing since an exact match just requires to store the bytes it needs to be checked against. The encoding logic is dead simple: it simply returns the underlying bytes slice. The decoding is a bit more involved: it compares the input slice to the underlying ExactMatch one and returns an error if it’s not a match.\nThe last line is a compile time type check to make sure ExactMatch actually implements the Codec interface. Now for the hexadecimal box:\n\/\/ codecs.go [...] type HexadecimalUInt8 struct { Variable *uint8 } func (h HexadecimalUInt8) Encode() ([]byte, error) { return []byte(fmt.Sprintf("%02x", uint64(*h.Variable))), nil } func (h HexadecimalUInt8) Decode(input []byte) ([]byte, error) { if len(input) < 2 { return nil, fmt.Errorf("expected at least 2 bytes") } n, err := strconv.ParseUint(string(input[:2]), 16, 8) if err != nil { return nil, err } *h.Variable = uint8(n) return input[2:], nil } var _ Codec = HexadecimalUInt8{} I can’t use type aliasing here as we need a way to store a pointer to the “tied” variable.\nEncoding and decoding are managed thanks to standard lib functions. Maybe there are more performant options but this will do for now.\nThe termination box’s code is even easier:\n\/\/ codecs.go [...] type NoMoreBytes struct{} func (n NoMoreBytes) Encode() ([]byte, error) { return nil, nil } func (n NoMoreBytes) Decode(input []byte) ([]byte, error) { if len(input) != 0 { return nil, fmt.Errorf("expected no more bytes") } return input, nil } var _ Codec = NoMoreBytes{} Using struct {} makes this abstraction’s size 0 which is convenient.\nWe got all our boxes but we still need a way to link them together:\n\/\/ combinators.go package godec type Sequence []Codec func (s Sequence) Encode() ([]byte, error) { var out []byte for _, codec := range s { codecOut, err := codec.Encode() if err != nil { return nil, err } out = append(out, codecOut...) } return out, nil } func (s Sequence) Decode(input []byte) (remaining []byte, err error) { remaining = input for _, codec := range s { remaining, err = codec.Decode(remaining) if err != nil { return nil, err } } return remaining, nil } A Sequence Encodes by applying its list of Codecs one after the other. Decoding works the same way.\nAll our components are ready, we can now compose them in a color Codec:\n\/\/ examples\/colors\/codec.go package colors import "github.com\/juliendoutre\/godec" func Codec(red, green, blue *uint8) godec.Sequence { return godec.Sequence([]godec.Codec{ godec.ExactMatch([]byte("#")), godec.HexadecimalUInt8{Variable: red}, godec.HexadecimalUInt8{Variable: green}, godec.HexadecimalUInt8{Variable: blue}, godec.NoMoreBytes{}, }) } And finally we can test it for simple cases:\n\/\/ examples\/colors\/codec_test.go package colors_test import ( "testing" "github.com\/juliendoutre\/godec\/examples\/colors" "github.com\/stretchr\/testify\/assert" ) func TestColorEncoding(t *testing.T) { red := uint8(13) green := uint8(89) blue := uint8(42) codec := colors.Codec(&red, &green, &blue) out, err := codec.Encode() assert.NoError(t, err) assert.Equal(t, []byte("#0d592a"), out) } func TestValidColorDecoding(t *testing.T) { var red, green, blue uint8 codec := colors.Codec(&red, &green, &blue) remainder, err := codec.Decode([]byte("#3399ff")) assert.NoError(t, err) assert.Equal(t, 0, len(remainder)) assert.Equal(t, uint8(51), red) assert.Equal(t, uint8(153), green) assert.Equal(t, uint8(255), blue) } I use the https:\/\/github.com\/stretchr\/testify library for testing. I added it to the project dependencies with go get -u github.com\/stretchr\/testify\/assert. Nice, it works \ud83c\udf89\nProperty based testing #Testing parsers for some values is nice but it does not provide a great coverage… What about weird corner cases? Will our code handle them just fine?\nOne approach to cover more test cases is property based testing. The goal is to generate test inputs at random, feed them to our code and then check some property is observed for all outputs. In the case of a codec, one simple property we wanna respect is that decoding is the invert function of encoding.\nIt happens golang provides a property based testing framework without any dependency required! It’s based on the QuickCheck library and can be leveraged as simply as:\n\/\/ examples\/colors\/codec_test.go [...] func TestInversibleProperty(t *testing.T) { f := func(expectedRed, expectedGreen, expectedBlue uint8) bool { actualRed := expectedRed actualGreen := expectedGreen actualBlue := expectedBlue encoder := colors.Codec(&expectedRed, &expectedGreen, &expectedBlue) out, err := encoder.Encode() if err != nil { return false } decoder := colors.Codec(&actualRed, &actualGreen, &actualBlue) remainder, err := decoder.Decode(out) if err != nil { return false } return len(remainder) == 0 && expectedRed == actualRed && expectedGreen == actualGreen && expectedBlue == actualBlue } if err := quick.Check(f, &quick.Config{}); err != nil { t.Error(err) } } The quick.Check(f, &quick.Config{}) function is where the magic happens. The library generates random inputs with types matching the f function’s signature, call it with those arguments and check it returns true or raises an error.\nA more involved use case: URLs #Wikipedia does a great job at summarizing the URL format: https:\/\/en.wikipedia.org\/wiki\/URL#Syntax. It even provides a syntax diagram which looks a lot like our state machine from the previous section: Picture by Alhadis on Wikipedia, under CC BY-SA 4.0 If we start writing a codec using our existing blocks we find ourselves pretty limited:\n\/\/ examples\/url\/codec.go package url import "github.com\/juliendoutre\/godec" func Codec(scheme, username, password, host *string, port *uint, path *string, query, fragment *string) godec.Sequence { return godec.Sequence([]godec.Codec{ \/\/ TODO: parse scheme godec.ExactMatch([]byte(":")), \/\/ TODO: parse path \/\/ TODO: parse eventual query \/\/ TODO: parse eventual fragment godec.NoMoreBytes{}, }) } We can start by implementing the scheme parser. It’s pretty specific to URL parsing, so let’s simply keep it as an unexported struct in the examples\/url package:\n\/\/ examples\/url\/scheme.go package url import ( "fmt" "github.com\/juliendoutre\/godec" ) type Scheme struct { scheme *string } func (s Scheme) Encode() ([]byte, error) { for _, c := range []byte(*s.scheme) { if !(c == '+') && !(c == '.') && !(c == '-') && !isASCIIDigit(c) && !isASCIILetter(c) { return nil, fmt.Errorf("invalid character %q", c) } } return []byte(*s.scheme), nil } func (s Scheme) Decode(input []byte) ([]byte, error) { \/\/ Reject empty schemes. if len(input) == 0 { return nil, fmt.Errorf("expected a scheme") } \/\/ See https:\/\/datatracker.ietf.org\/doc\/html\/rfc1738#section-2.1: \/\/ Scheme names consist of a sequence of characters. The lower case \/\/ letters "a"--"z", digits, and the characters plus ("+"), period \/\/ ("."), and hyphen ("-") are allowed. For resiliency, programs \/\/ interpreting URLs should treat upper case letters as equivalent to \/\/ lower case in scheme names (e.g., allow "HTTP" as well as "http"). i := 0 for i = 0; i < len(input); i++ { if !(input[i] == '+') && !(input[i] == '.') && !(input[i] == '-') && !isASCIIDigit(input[i]) && !isASCIILetter(input[i]) { break } } if i == 0 { return nil, fmt.Errorf("expected a scheme") } *s.scheme = string(input[:i]) return input[i:], nil } var _ godec.Codec = Scheme{} Let’s write a dead simple test to validate decoding is working:\n\/\/ examples\/url\/codec_test.go package url_test import ( "testing" "github.com\/juliendoutre\/godec\/examples\/url" "github.com\/stretchr\/testify\/assert" ) func TestSchemeValidDecoding(t *testing.T) { var scheme string decoder := url.Codec(&scheme, nil, nil, nil, nil, nil, nil, nil) remainder, err := decoder.Decode([]byte("http:")) assert.NoError(t, err) assert.Equal(t, 0, len(remainder)) assert.Equal(t, "http", scheme) } Implementing the other parsers (check out the complete code at https:\/\/github.com\/juliendoutre\/godec\/tree\/main\/examples\/url) is pretty similar, in the end we can write more complete tests:\n\/\/ examples\/url\/codec_test.go package url_test import ( "testing" "github.com\/juliendoutre\/godec\/examples\/url" "github.com\/stretchr\/testify\/assert" ) func TestSchemeValidHTTPURLDecoding(t *testing.T) { var scheme string var username string var password string var host string var port uint var path string var query string var fragment string decoder := url.Codec(&scheme, &username, &password, &host, &port, &path, &query, &fragment) remainder, err := decoder.Decode([]byte("http:\/\/google.com:443\/test?page=1#title")) assert.NoError(t, err) assert.Equal(t, 0, len(remainder)) assert.Equal(t, "http", scheme) assert.Equal(t, "", username) assert.Equal(t, "", password) assert.Equal(t, "google.com", host) assert.Equal(t, uint(443), port) assert.Equal(t, "\/test", path) assert.Equal(t, "page=1", query) assert.Equal(t, "title", fragment) } func TestSchemeValidPostgresDecoding(t *testing.T) { var scheme string var username string var password string var host string var port uint var path string var query string var fragment string decoder := url.Codec(&scheme, &username, &password, &host, &port, &path, &query, &fragment) remainder, err := decoder.Decode([]byte("postgres:\/\/user:password@localhost:5432\/database")) assert.NoError(t, err) assert.Equal(t, 0, len(remainder)) assert.Equal(t, "postgres", scheme) assert.Equal(t, "user", username) assert.Equal(t, "password", password) assert.Equal(t, "localhost", host) assert.Equal(t, uint(5432), port) assert.Equal(t, "\/database", path) assert.Equal(t, "", query) assert.Equal(t, "", fragment) } I’m not going to expand too much on this but there’s probably room for improvements and it could definitely use some more testing!\nConclusion #And here we go, we wrote a simple Go parser library with very basic primitives and are able to use it to parse two data formats: hexadecimal color codes and URLs.\nI intentionally did not build a lot of basic blocks. When you have a look at https:\/\/github.com\/rust-bakery\/nom it provides many utils but in Go, abstraction (understand interfaces) has a cost.\nOne next step could be supporting more complex formats such as JSON but this will be for another day!\nSee you next time \ud83d\udc4b\n","date":"2024-04-03","permalink":"https:\/\/juliendoutre.github.io\/posts\/lets-write-a-parsing-library-in-go\/","section":"Posts","summary":"Walkthrough of an encoding\/decoding library implementation","title":"Let's write a parsing library in Go!"},{"content":"I wanted to have my own blog for a long time. Finally, some day last week, I found the time to set up one. This article will reflect on my thought process and the stack I ended up choosing.\nMy requirements were pretty straigthforward. I wanted:\nto write posts using just markdown. I’m not a fan of fancy text editors, they’re not reactive enough. Being able to write in my IDE (vscode) where I already manage my code and my terminal makes it easier for me to gather all my tools at the same place. to host the website myself, not just posting on Medium or Wordpress for instance. I wanted to be able to control the very content I would expose publically from the articles’ body to the pages’ HTML. the solution to be as cheap as possible. Do I really need to elaborate? Additionally, I did not want to build something from scratch, especially not relying on JS frameworks like React which are way too overkill for small projects. Moreover I’m not a frontend engineer, I’m really bad at drawing with CSS (I still struggle centering <div> inside flex boxes, how do these guys have their tags behave as they want them to?! \ud83e\udd2f).\nNaturally, the go-to for static websites supporting markdown is Hugo. It did not take me a lot of time to realize it was a perfect fit for my use case:\nbrew install hugo hugo new site juliendoutre.github.io and I had my project initialized.\nhugo serve and I had a locally version of the website running.\nWhat took me the most time to figure out was obviously (and as foresaw my friend Edouard) which theme to use.\nI had vague memories of the Gitbook project which I liked the chapter-based organization with a nice vertical left side panel. I checked their website and noticed they stopped supporting their self-hosted solution to focus on a SAAS platform. Too bad \ud83e\udd37 Plus it did not really fit a blog post use case, missing a lot of features I could find elsewhere.\nAfter exploring the Hugo theme showcase I ended up choosing the Congo theme which was appealing to me for the following reasons:\nit has a search bar it has a dark mode the home page can show as a profile page configuration options are crystal clear it uses Tailwind and I heard it’s the cool new thing. The only non-trivial customization was changing the default flaticon. I had to lookup some other persons’ blogs source code to find the right file names to use in the static folder of my project.\nRegarding the hosting, I considered for a second deploying everything in my AWS account using S3 and CloudFront. But in the end, GitHub pages is easier to use, integrates seamlessly with GitHub actions, plus it’s free (I don’t even have to pay for a domain name)!\nMy current development workflow is pushing to the main branch, waiting for the GitHub action to generate the public folder to the gh-pages branch, which is then served by GitHub pages on the domain matching the repository’s name. I’ll see how this can be improved in the future!\nIn the meantime, if you’re curious, you can check this website source code at https:\/\/github.com\/juliendoutre\/juliendoutre.github.io \ud83e\udd13\nSee you next time \ud83d\udc4b\n","date":"2023-04-03","permalink":"https:\/\/juliendoutre.github.io\/posts\/how-i-built-this-website\/","section":"Posts","summary":"An explanation of how this website is built and served","title":"How I built this website"},{"content":"This is the first post on this website!\n","date":"2023-03-31","permalink":"https:\/\/juliendoutre.github.io\/posts\/hello-world\/","section":"Posts","summary":"This is my first post on this site!","title":"Hello World!"},{"content":"","date":null,"permalink":"https:\/\/juliendoutre.github.io\/categories\/","section":"Categories","summary":"","title":"Categories"},{"content":"Hacking and coding challenges I’ve been working on.\n","date":null,"permalink":"https:\/\/juliendoutre.github.io\/challenges\/","section":"Challenges","summary":"","title":"Challenges"},{"content":"Books I enjoyed reading and recommend checking out!\n","date":null,"permalink":"https:\/\/juliendoutre.github.io\/readings\/","section":"Readings","summary":"","title":"Readings"},{"content":"","date":null,"permalink":"https:\/\/juliendoutre.github.io\/tags\/","section":"Tags","summary":"","title":"Tags"}]