{"@attributes":{"version":"2.0"},"channel":{"title":"Tom's Blog","link":"https:\/\/tomaugspurger.net\/","description":"Recent content on Tom's Blog","generator":"Hugo -- 0.147.0","language":"en-us","lastBuildDate":"Sun, 28 Dec 2025 15:00:24 -0600","item":[{"title":"My 2025 Year in Books","link":"https:\/\/tomaugspurger.net\/posts\/books-2025\/","pubDate":"Sun, 28 Dec 2025 15:00:24 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/books-2025\/","description":"

It’s time for another review for books I read this year (previously: 2024<\/a>, 2022<\/a>). According to my GoodReads<\/a>, I read 27 books this year.\nHere are some highlights:<\/p>\n

The Demon-Haunted World<\/h2>\n

<\/a><\/p>\n

I started the year with Carl Sagan’s The Demon-Haunted World<\/em>, as some kind of antidote to current \/ coming events.\nI last read this in about 2010, and held it in very high regard. I still do, but Carl comes off as a bit<\/em> of a fuddy duddy at times (especially when talking about “the youth” today \/ in the 1990s). That’s not to say that he’s wrong about where society has gone (quiet the opposite), but it as a kind of tone. If you’re interested in an introduction to skepticism, I’d probably recommend the Skeptic’s Guide to the Universe<\/em><\/a>.<\/p>"},{"title":"GPU-Accelerated Zarr","link":"https:\/\/tomaugspurger.net\/posts\/gpu-accelerated-zarr\/","pubDate":"Thu, 11 Dec 2025 08:00:00 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/gpu-accelerated-zarr\/","description":"

This post gives detailed background to my PyData Global talk, “GPU-Accelerated\nZarr” (slides<\/a>, video<\/a>). It\ndeliberately gets into the weeds, but I will try to provide some background for\npeople who are new to Zarr, GPUs, or both.<\/p>\n

The first takeaway is that zarr-python natively supports<\/a>\nNVIDIA GPUs. With a one-line zarr.config.enable_gpu()<\/code> you can configure zarr\nto return CuPy arrays, which reside on your GPU:<\/p>\n

>>><\/span> import<\/span> zarr\n<\/span><\/span>>>><\/span> zarr.<\/span>config.<\/span>enable_gpu()\n<\/span><\/span>>>><\/span> z =<\/span> zarr.<\/span>open_array("path\/to\/store.zarr"<\/span>, mode=<\/span>"r"<\/span>)\n<\/span><\/span>>>><\/span> type(z[:])\n<\/span><\/span>cupy.<\/span>ndarray\n<\/span><\/span><\/code><\/pre><\/div>
The second takeaway, and the main focus of this post, is that that simple\none-liner leaves performance on the table. It depends a bit on your workload,\nbut I’d claim that Zarr’s data loading pipeline shouldn’t<\/em> ever be the\nbottleneck. Achieving maximum throughput today requires some care to ensure that\nthe system’s resources are used efficiently. I’m hopeful that we can improve the\nlibraries to do the right thing in more situations.<\/p>"},{"title":"Gone Sailing","link":"https:\/\/tomaugspurger.net\/posts\/gone-sailing\/","pubDate":"Sat, 18 Oct 2025 07:40:38 -0500","guid":"https:\/\/tomaugspurger.net\/posts\/gone-sailing\/","description":"
First, some context: in August I attended the Saylorville Yatch Club Sailing\nSchool<\/a> Adult Small Boat class. This is a 3-day course that mixes some\ntime in the classroom learning the theory and jargon (so much jargon!) with a\nbunch of time on the water. I had a bit of experience from sailing on summer\nweekends with my family growing up, but I wanted to learn more before going out\non my own.<\/p>"},{"title":"Cloud Native Geospatial Conference (2025)","link":"https:\/\/tomaugspurger.net\/posts\/cng-forum-2025\/","pubDate":"Sun, 04 May 2025 00:00:00 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/cng-forum-2025\/","description":"
You can watch a video version of this talk at https:\/\/youtu.be\/BFFHXNBj7nA<\/a><\/em><\/p>\n
On Thursday, I presented a talk, GPU Accelerated Cloud-Native Geospatial<\/em>, at the\ninaugural Cloud-Native Geospatial Conference (slides<\/a> here). This post will\ngive an overview of the talk and some background on the prep. But first I wanted\nto say a bit about the conference itself.<\/p>\n
The organizers (Michelle Roby, Jed Sundell, and others from Radiant Earth) did a\nfantastic job putting on the event. I only have the smallest experience with\nhelping run a conference, but I know it’s a ton of work. They did a great job\nhosting this first run of conference.<\/p>"},{"title":"High Performance Remote IO","link":"https:\/\/tomaugspurger.net\/posts\/high-performance-remote-io\/","pubDate":"Fri, 28 Feb 2025 15:18:34 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/high-performance-remote-io\/","description":"
I have a new post up at the NVIDIA technical blog on High-Performance Remote IO with NVIDIA KvikIO<\/a>.1<\/a><\/sup><\/p>\n
This is mostly general-purpose advice on getting good performance out of cloud\nobject stores (I guess I can’t get away from them), but has some specifics for\npeople using NVIDIA GPUs.<\/p>\n
\n
In the RAPIDS context, NVIDIA KvikIO is notable because<\/p>\n
    \n
  1. It automatically chunks large requests into multiple smaller ones and makes those requests concurrently.<\/li>\n
  2. It can read efficiently into host or device memory, especially if GPU Direct Storage is enabled.<\/li>\n
  3. It\u2019s fast.<\/li>\n<\/ol><\/blockquote>\n
    As part of preparing this, I got to write some C++. Not a fan!<\/p>"},{"title":"Iowa's Proposed State Science Standards","link":"https:\/\/tomaugspurger.net\/posts\/iowa-academic-standards-2025\/","pubDate":"Sat, 01 Feb 2025 12:00:00 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/iowa-academic-standards-2025\/","description":"
    My local Department of Education has a public comment\nperiod<\/a>\nfor some proposed changes to Iowa’s science education standards. If you live in\nIowa, I’d encourage you to read the\nproposal<\/a> (PDF) and share\nfeedback through the survey<\/a>. If you,\nlike me, get frustrated with how difficult it is to see what’s changed or link\nto a specific piece of text, read on.<\/p>\n
    I’d heard rumblings that there were some controversial changes around evolution\nand climate change. But rather than just believing what I read in a headline, I\ndecided to do my own research (science in action, right?).<\/p>"},{"title":"Dask Geopandas Spatial Shuffle","link":"https:\/\/tomaugspurger.net\/posts\/dask-geopandas-spatial-shuffle\/","pubDate":"Wed, 18 Dec 2024 10:33:34 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/dask-geopandas-spatial-shuffle\/","description":"
    Over at https:\/\/github.com\/opengeospatial\/geoparquet\/discussions\/251<\/a>, we’re\nhaving a nice discussion about how best to partition geoparquet files for\nserving over object storage. Thanks to geoparquet’s design, just being an\nextension of parquet, it immediately benefits from all the wisdom around how\nbest to partition plain parquet datasets. The only additional wrinkle for\ngeoparquet is, unsurprisingly, the geo component.<\/p>\n
    It’s pretty common for users to read all the features in a small spatial area (a\ncity, say) so optimizing for that use case is a good default. Simplifying a bit,\nreading small spatial subsets of a larger dataset will be fastest if all the\nfeatures that are geographically close together are also “close” together in the\nparquet dataset, and each part of the parquet dataset only contains data that’s\nphysically close together. That gives you the data you want in the fewest number\nof file reads \/ HTTP requests, and minimizes the amount of “wasted” reads (data\nthat’s read, only to be immediately discarded because it’s outside your area of\ninterest).<\/p>"},{"title":"My 2024 Year in Books","link":"https:\/\/tomaugspurger.net\/posts\/books-2024\/","pubDate":"Wed, 11 Dec 2024 08:00:00 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/books-2024\/","description":"
    Here’s another Year in Books (I missed last year, but here’s 2022<\/a>).<\/p>\n
    Most of these came from recommendations by friends, The Incomparable<\/em>’s Book\nClub<\/a> and (a new\nsource), the “Books in the Box” episodes of Oxide and\nFriends<\/a>.<\/p>\n
    The Soul of a New Machine<\/em>, by Tracy Kidder<\/h2>\n
    <\/a><\/p>\n
    I technically read it in the last few days of 2023, but included here because I\nliked it so much. This came recommended by the Oxide and Friends podcast’s\nBooks in the Box<\/a>\nepisode. I didn’t know a ton about the history of computing, but have been\npicking up an appreciation for it thanks to reading this book. It goes into a\nton of detail about what it took Data General to design and release a new\nmachine. Highly recommended to anyone interested in computing.<\/p>"},{"title":"Serializing Dataclasses","link":"https:\/\/tomaugspurger.net\/posts\/serializing-dataclasses\/","pubDate":"Sat, 31 Aug 2024 12:00:00 -0500","guid":"https:\/\/tomaugspurger.net\/posts\/serializing-dataclasses\/","description":"
    This post is a bit of a tutorial on serializing and deserializing Python\ndataclasses<\/a>. I’ve been hacking on zarr-python-v3<\/a> a bit, which uses some\ndataclasses to represent some metadata objects. Those objects need to be\nserialized to and deserialized from JSON.<\/p>\n
    This is a (surprisingly?) challenging area, and there are several excellent\nlibraries out there that you should probably use. My personal favorite is\nmsgspec<\/a>, but cattrs<\/a>, pydantic<\/a>, and pyserde<\/a> are also options. But\nhopefully this can be helpful for understanding how those libraries work at a\nconceptual level (their exact implementations will look very different.) In\nzarr-python’s case, this didn’t quite<\/em> warrant needing to bring in a\ndependency, so we rolled our own.<\/p>"},{"title":"stac-geoparquet","link":"https:\/\/tomaugspurger.net\/posts\/stac-geoparquet\/","pubDate":"Thu, 29 Aug 2024 20:00:00 -0500","guid":"https:\/\/tomaugspurger.net\/posts\/stac-geoparquet\/","description":"
    I wrote up a quick introduction to stac-geoparquet<\/a> on the Cloud Native Geo blog with Kyle Barron<\/a> and Chris Holmes<\/a>.<\/p>\n
    The key takeaway:<\/p>\n
    \n
    STAC GeoParquet offers a very convenient and high-performance way to distribute large STAC collections, provided the items in that collection are pretty homogenous<\/p><\/blockquote>\n
    Check out the project at http:\/\/github.com\/stac-utils\/stac-geoparquet<\/a>.<\/p>"},{"title":"What's Next? (2024 edition)","link":"https:\/\/tomaugspurger.net\/posts\/leaving-microsoft\/","pubDate":"Mon, 12 Aug 2024 07:00:00 -0500","guid":"https:\/\/tomaugspurger.net\/posts\/leaving-microsoft\/","description":"
    I have, as they say, some personal news to share. On Monday I (along with some very<\/em> talented teammates, see below<\/a> if you’re hiring) was laid off from Microsoft as part of a reorganization. Like my Moving to Microsoft<\/a> post, I wanted to jot down some of the things I got to work on.<\/p>\n
    For those of you wondering, the Planetary Computer<\/a> project does<\/em> continue, just without me.<\/p>\n
    Reflections<\/h2>\n
    It should go without saying that all<\/em> of this was a team effort. I’ve been incredibly fortunate to have great teammates over the years, but the team building out the Planetary Computer<\/a> was especially fantastic.\nJust like before, this will be very self-centered and project-focused, overlooking all the other people and work that went into this.<\/p>"},{"title":"My Real-World Match \/ Case","link":"https:\/\/tomaugspurger.net\/posts\/my-real-world-match\/","pubDate":"Wed, 13 Dec 2023 21:00:00 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/my-real-world-match\/","description":"
    Ned Batchelder recently shared Real-world match\/case<\/a>, showing a real example of Python’s Structural Pattern Matching<\/a>. These real-world examples are a great complement to the tutorial, so I’ll share mine.<\/p>\n
    While working on some STAC + Kerchunk stuff<\/a>, in this pull request<\/a> I used the match statement to parse some nested objects:<\/p>\n
    for<\/span> k, v in<\/span> refs.<\/span>items():\n<\/span><\/span> match<\/span> k.<\/span>split("\/"<\/span>):\n<\/span><\/span> case<\/span> [".zgroup"<\/span>]:\n<\/span><\/span> # k = ".zgroup"<\/span>\n<\/span><\/span> item.<\/span>properties["kerchunk:zgroup"<\/span>] =<\/span> json.<\/span>loads(v)\n<\/span><\/span> case<\/span> [".zattrs"<\/span>]:\n<\/span><\/span> # k = ".zattrs"<\/span>\n<\/span><\/span> item.<\/span>properties["kerchunk:zattrs"<\/span>] =<\/span> json.<\/span>loads(v)\n<\/span><\/span> case<\/span> [variable, ".zarray"<\/span>]:\n<\/span><\/span> # k = "prcp\/.zarray"<\/span>\n<\/span><\/span> if<\/span> u :=<\/span> item.<\/span>properties["cube:dimensions"<\/span>].<\/span>get(variable):\n<\/span><\/span> u["kerchunk:zarray"<\/span>] =<\/span> json.<\/span>loads(refs[k])\n<\/span><\/span> elif<\/span> u :=<\/span> item.<\/span>properties["cube:variables"<\/span>].<\/span>get(variable):\n<\/span><\/span> u["kerchunk:zarray"<\/span>] =<\/span> json.<\/span>loads(refs[k])\n<\/span><\/span> case<\/span> [variable, ".zattrs"<\/span>]:\n<\/span><\/span> # k = "prcp\/.zattrs"<\/span>\n<\/span><\/span> if<\/span> u :=<\/span> item.<\/span>properties["cube:dimensions"<\/span>].<\/span>get(variable):\n<\/span><\/span> u["kerchunk:zattrs"<\/span>] =<\/span> json.<\/span>loads(refs[k])\n<\/span><\/span> elif<\/span> u :=<\/span> item.<\/span>properties["cube:variables"<\/span>].<\/span>get(variable):\n<\/span><\/span> u["kerchunk:zattrs"<\/span>] =<\/span> json.<\/span>loads(refs[k])\n<\/span><\/span> case<\/span> [variable, index]:\n<\/span><\/span> # k = "prcp\/0.0.0"<\/span>\n<\/span><\/span> if<\/span> u :=<\/span> item.<\/span>properties["cube:dimensions"<\/span>].<\/span>get(variable):\n<\/span><\/span> u.<\/span>setdefault("kerchunk:value"<\/span>, collections.<\/span>defaultdict(dict))\n<\/span><\/span> u["kerchunk:value"<\/span>][index] =<\/span> refs[k]\n<\/span><\/span> elif<\/span> u :=<\/span> item.<\/span>properties["cube:variables"<\/span>].<\/span>get(variable):\n<\/span><\/span> u.<\/span>setdefault("kerchunk:value"<\/span>, collections.<\/span>defaultdict(dict))\n<\/span><\/span> u["kerchunk:value"<\/span>][index] =<\/span> refs[k]\n<\/span><\/span><\/code><\/pre><\/div>
    I wanted to share an update on a couple of developments in the STAC<\/a> ecosystem\nthat I’m excited about. It’s a great sign that even after 2 years after its\ninitial release, the STAC ecosystem is still growing and improving how we can\ncatalog, serve, and access geospatial data.<\/p>\n
    STAC and Geoparquet<\/h2>\n
    A STAC API is a great way to query for data. But, like any API serving JSON, its\nthroughput is limited. So in May 2022, the Planetary Computer team decided to\nexport snapshots of our STAC database as geoparquet<\/a>. Each STAC collection is\nexported as a Parquet dataset, where each record in the dataset is a STAC item.\nWe pitched this as a way to do bulk queries<\/a> over the data, where\nreturning many and many pages of JSON would be slow (and expensive for our\nservers and database).<\/p>"},{"title":"Gone Rafting","link":"https:\/\/tomaugspurger.net\/posts\/gone-rafting\/","pubDate":"Sun, 13 Aug 2023 14:30:19 -0500","guid":"https:\/\/tomaugspurger.net\/posts\/gone-rafting\/","description":"
    Last week, I was fortunate to attend Dave Beazley’s Rafting\nTrip<\/a> course. The pretext of the course is to\nimplement the Raft Consensus Algorithm<\/a>.<\/p>\n
    I’ll post more about Raft, and the journey of implementing, it later. But in\nbrief, Raft is an algorithm that lets a cluster of machines work together to\nreliably<\/em> do something. If you had a service that needed to stay up (and stay\nconsistent), even if some of the machines in the cluster went down, then you\nmight want to use Raft.<\/p>"},{"title":"National Water Model on Azure","link":"https:\/\/tomaugspurger.net\/posts\/noaa-nwm\/","pubDate":"Thu, 25 May 2023 12:04:06 -0500","guid":"https:\/\/tomaugspurger.net\/posts\/noaa-nwm\/","description":"
    A few colleagues and I recently presented at the CIROH Training and Developers Conference<\/a>.\nIn preparation for that I created a Jupyter Book<\/a>. You can view it at https:\/\/tomaugspurger.net\/noaa-nwm\/intro.html<\/a>\nI created a few cloud-optimized versions for subsets of the data, but those will be going away since we don’t have operational pipelines to keep them up to date. But hopefully the static notebooks are still helpful.<\/p>\n
    Lessons learned<\/h2>\n
    Aside from running out of time (I always prepare too much material for the amount of time), I think things went well. JupyterHub (perhaps + Dask) and Kubernetes continues to be a great way to run a workshop.<\/p>"},{"title":"Jupyter, STAC, and Tool Building","link":"https:\/\/tomaugspurger.net\/posts\/jupyter-stac-browser\/","pubDate":"Sat, 15 Apr 2023 08:00:00 -0500","guid":"https:\/\/tomaugspurger.net\/posts\/jupyter-stac-browser\/","description":"
    Over in Planetary Computer land, we’re working on<\/a> bringing Sentinel-5P<\/a> into our STAC catalog.<\/p>\n
    STAC items require a geometry<\/code> property, a GeoJSON object that describes the footprint of the assets. Thanks to the satellites’ orbit and the (spatial) size of the assets, we started with some…interesting… footprints:<\/p>\n
    <\/p>\n
    That initial footprint, shown in orange, would render the STAC collection essentially useless for spatial searches. The assets don’t actually cover (most of) the southern hemisphere.<\/p>"},{"title":"py-spy in Azure Batch","link":"https:\/\/tomaugspurger.net\/posts\/azure-batch-pyspy\/","pubDate":"Wed, 22 Feb 2023 15:11:37 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/azure-batch-pyspy\/","description":"
    Today, I was debugging a hanging task in Azure Batch<\/a>.\nThis short post records how I used py-spy<\/a> to investigate the problem.<\/p>\n
    Background<\/h2>\n
    Azure Batch is a compute service that we use to run container\nworkloads<\/a>.\nIn this case, we start up a container that processes a bunch of GOES-GLM data to\ncreate STAC items<\/a> for the Planetary\nComputer<\/a> . The workflow is essentially\na big<\/p>\n
    for<\/span> url in<\/span> urls:\n<\/span><\/span> local_file =<\/span> download_url(url)\n<\/span><\/span> stac.<\/span>create_item(local_file)\n<\/span><\/span><\/code><\/pre><\/div>
    The Planetary Computer made its January 2023 release<\/a> a couple weeks back.<\/p>\n
    The flagship new feature is a really<\/em> cool new ability to visualize the Microsoft AI-detected Buildings Footprints<\/a> dataset. Here’s a little demo made by my teammate, Rob:<\/p>\n
    \nYour browser doesn't support HTML video. Here is a\nlink to the video<\/a> instead.\n<\/p>\n<\/video>\n
    Currently, enabling this feature required converting the data from its native geoparquet<\/a> to a lot<\/em> of protobuf files with Tippecanoe<\/a>. I’m very excited about projects to visualize the geoparquet data directly (see Kyle Barron’s demo<\/a>) but for now we needed to do the conversion.<\/p>"},{"title":"Cloud Optimized Vibes","link":"https:\/\/tomaugspurger.net\/posts\/cloud-optimized-quandary\/","pubDate":"Sat, 14 Jan 2023 16:16:11 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/cloud-optimized-quandary\/","description":"
    Over on the Planetary Computer<\/a> team, we get to have a lot of fun discussions about doing geospatial data analysis on the cloud. This post summarizes some work we did, and the (I think) interesting conversations that came out of it.<\/p>\n
    Background: GOES-GLM<\/h2>\n
    The instigator in this case was onboarding a new dataset to the Planetary Computer, GOES-GLM<\/a>. GOES is a set of geostationary weather satellites operated by NOAA, and GLM is the Geostationary Lightning Mapper, an instrument on the satellites that’s used to monitor lightning. It produces some really neat (and valuable) data.<\/p>"},{"title":"Queues in the News","link":"https:\/\/tomaugspurger.net\/posts\/queues\/","pubDate":"Mon, 26 Dec 2022 13:35:24 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/queues\/","description":"
    I came across a couple of new (to me) uses of queues recently. When I came up with the title to this article I knew I had to write them up together.<\/p>\n
    Queues in Dask<\/h2>\n
    Over at the Coiled Blog<\/a>, Gabe Joseph has a nice post summarizing a huge amount of effort addressing a problem that’s been vexing demanding Dask users for years. The main symptom of the problem was unexpectedly high memory usage on workers, leading to crashing workers (which in turn caused even more network communication, and so more memory usage, and more crashing workers). This is actually a problem I worked on a bit back in 2019, and I made very little progress.<\/p>"},{"title":"My 2022 Year in Books","link":"https:\/\/tomaugspurger.net\/posts\/books-2022\/","pubDate":"Wed, 21 Dec 2022 07:25:47 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/books-2022\/","description":"
    It’s “Year in X” time, and here’s my 2022 Year in Books<\/a> on GoodReads. I’ll cover some highlights here.<\/p>\n
    Many of these recommendations came from the Incomparable’s Book Club<\/a>, part of the main Incomparable podcast. In particular, episode 600 The Machine was a Vampire<\/a> which is a roundup of their favorites from the 2010s.<\/p>\n
    Bookended by Murderbot Diaries<\/h2>\n\n
    I started and ended this year (so far) with a couple installments in the Murderbot Diaries<\/a>. These follow a robotic \/ organic “Security Unit” that’s responsible for taking care of humans in dangerous situations. We pick up after an unfortunate incident where it seems to have gone rouge and murdered her clients (hence, the murderbot) and hacked its governor module to essentially become “free”.<\/p>"},{"title":"Podcast: Revolutions","link":"https:\/\/tomaugspurger.net\/posts\/revolutions\/","pubDate":"Tue, 20 Dec 2022 16:56:57 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/revolutions\/","description":"
    Mike Duncan is wrapping up his excellent Revolutions<\/a> podcast. If you’re at all interested in history then now is a great time to pick it up. He takes the concept of “a revolution” and looks at it through the lens of a bunch of revolutions throughout history. The appendix episodes from the last few weeks have really tied things together, looking at whats common (and not) across all the revolutions covered in the series.<\/p>"},{"title":"Rebooting","link":"https:\/\/tomaugspurger.net\/posts\/rebooting\/","pubDate":"Sun, 18 Dec 2022 16:51:46 -0600","guid":"https:\/\/tomaugspurger.net\/posts\/rebooting\/","description":"
    Like some others<\/a>, I’m getting back into blogging.<\/p>\n
    I’ll be “straying from my lane” and won’t just be writing about Python data libraries (though there will still be some of that).\nIf you too would like to blog more, I’d encourge you to read Simon Willison’s What to blog About<\/a> and Matt Rocklin’s Write Short Blogposts<\/a>.<\/p>\n
    Because I’m me, I couldn’t just make a new post. I also had to switch static site generators, just becauase. All the old links, including my RSS feed<\/a>, should continue to work. If you spot any issues, let me know<\/a> (I think I’ve fixed at least one bug in the RSS feed, apologies for any spurious updates. But just in case, you might want to update your RSS links to http:\/\/tomaugspurger.net\/index.xml<\/a>).<\/p>"},{"title":"What's Next?","link":"https:\/\/tomaugspurger.net\/posts\/whats-next\/","pubDate":"Wed, 11 Nov 2020 00:00:00 +0000","guid":"https:\/\/tomaugspurger.net\/posts\/whats-next\/","description":"
    Some personal news: Last Friday was my last day at Anaconda<\/a>.\nNext week, I’m joining Microsoft’s AI for Earth<\/a> team. This is a very bittersweet transition. While I loved working at Anaconda and all the great people there, I’m extremely excited about what I’ll be working on at Microsoft.<\/p>\n
    Reflections<\/h2>\n
    I was inspired to write this section by Jim Crist’s post on a similar topic: https:\/\/jcristharif.com\/farewell-to-anaconda.html<\/a>.\nI’ll highlight some of the projects I worked on while at Anaconda. If you want to skip the navel gazing, skip down to what’s next<\/a>.<\/p>"},{"title":"Maintaining Performance","link":"https:\/\/tomaugspurger.net\/posts\/performance-regressions\/","pubDate":"Wed, 01 Apr 2020 00:00:00 +0000","guid":"https:\/\/tomaugspurger.net\/posts\/performance-regressions\/","description":"
    As pandas’ documentation<\/a> claims: pandas\nprovides high-performance<\/em> data structures. But how do we verify that the claim\nis correct? And how do we ensure that it stays<\/em> correct over many releases.\nThis post describes<\/p>\n
      \n
    1. pandas’ current setup for monitoring performance<\/li>\n
    2. My personal debugging strategy for understanding and fixing performance\nregressions when they occur.<\/li>\n<\/ol>\n
      I hope that the first section topic is useful for library maintainers and the\nsecond topic is generally useful for people writing performance-sensitive code.<\/p>"},{"title":"Compatibility Code","link":"https:\/\/tomaugspurger.net\/posts\/compatibility\/","pubDate":"Thu, 12 Dec 2019 00:00:00 +0000","guid":"https:\/\/tomaugspurger.net\/posts\/compatibility\/","description":"
      Compatibility Code<\/h1>\n
      Most libraries with dependencies will want to support multiple versions\nof that dependency. But supporting old version is a pain: it requires compatibility code<\/em>,\ncode that is around solely to get the same output from versions of a library. This post\ngives some advice on writing compatibility code.<\/p>\n
        \n
      1. Don’t write your own version parser<\/li>\n
      2. Centralize all version parsing<\/li>\n
      3. Use consistent version comparisons<\/li>\n
      4. Use Python’s argument unpacking<\/li>\n
      5. Clean up unused compatibility code<\/li>\n<\/ol>\n
        1. Don’t write your own version parser<\/h2>\n
        It can be tempting just do something like<\/p>"},{"title":"Dask Workshop","link":"https:\/\/tomaugspurger.net\/posts\/dask-workshop\/","pubDate":"Thu, 12 Dec 2019 00:00:00 +0000","guid":"https:\/\/tomaugspurger.net\/posts\/dask-workshop\/","description":"
        Dask Summit Recap<\/p>\n
        Last week was the first Dask Developer Workshop. This brought together many of\nthe core Dask developers and its heavy users to discuss the project. I want to\nshare some of the experience with those who weren’t able to attend.<\/p>\n
        This was a great event. Aside from any technical discussions, it was ncie to\nmeet all the people. From new acquaintences to people you’re on weekly calls\nwith, it was great to interact with everyone.<\/p>"},{"title":"pandas + binder","link":"https:\/\/tomaugspurger.net\/posts\/pandas-binder\/","pubDate":"Sun, 21 Jul 2019 00:00:00 +0000","guid":"https:\/\/tomaugspurger.net\/posts\/pandas-binder\/","description":"
        This post describes the start of a journey to get pandas’ documentation running\non Binder. The end result is this nice button:<\/p>\n
        \"Binder\"<\/a><\/p>\n
        \n
        For a while now I’ve been jealous of Dask’s examples\nrepository<\/a>. That’s a repository containing a\ncollection of Jupyter notebooks demonstrating Dask in action. It stitches\ntogether some tools to present a set of documentation that is both viewable\nas a static site at examples.dask.org<\/a>, and as a executable notebooks\non mybinder<\/a>.<\/p>"},{"title":"A Confluence of Extension","link":"https:\/\/tomaugspurger.net\/posts\/confluence-extension\/","pubDate":"Tue, 18 Jun 2019 00:00:00 +0000","guid":"https:\/\/tomaugspurger.net\/posts\/confluence-extension\/","description":"
        This post describes a few protocols taking shape in the scientific Python\ncommunity. On their own, each is powerful. Together, I think they enable for an\nexplosion of creativity in the community.<\/p>\n
        Each of the protocols \/ interfaces we’ll consider deal with extending.<\/p>\n
    We noticed that some Azure Batch tasks were hanging. Based on our logs, we knew\nit was somewhere in that for loop, but couldn’t determine exactly where things\nwere hanging. The goes-glm<\/a> stactools package we used does read a NetCDF file,\nand my experience with Dask biased me towards thinking the netcdf<\/code> library (or\nthe HDF5 reader it uses) was hanging. But I wanted to confirm that before trying\nto implement a fix.<\/p>"},{"title":"Planetary Computer Release: January 2023","link":"https:\/\/tomaugspurger.net\/posts\/pc-january-2023\/","pubDate":"Thu, 09 Feb 2023 00:00:00 +0000","guid":"https:\/\/tomaugspurger.net\/posts\/pc-january-2023\/","description":"
    The for<\/code> loop is iterating over a set of Kerchunk<\/a> references, which are essentially the keys for a Zarr<\/a> group. The keys vary a bit. They could be:<\/p>"},{"title":"STAC Updates I'm Excited About","link":"https:\/\/tomaugspurger.net\/posts\/stac-updates\/","pubDate":"Sun, 15 Oct 2023 12:00:00 -0500","guid":"https:\/\/tomaugspurger.net\/posts\/stac-updates\/","description":"
Last weekend I had the chance to sail in the 2025 Corn Coast Regatta<\/a>. I had\nsuch a great time that I had to jot down my thoughts before they fade. This\npost is mostly for (future) me. We’ll return to our regularly scheduled\nprogramming in a future post. I have a post on Zarr performance cooking.<\/p>\n