Getting started with Claude Code, on MacOS, with chruby and rspec

I’ve been writing ruby and rails for nearly 20 years. A couple weeks ago, I had gotten code snippets from copy-paste in a chat window, but I hadn’t even experimented with Claude Code or similar “can write code to your file system” tools.

I know some people are now using LLM’s to write all their code, which I’m not excited about, but I decided I couldn’t hold off any longer, and I had to at least understand how it worked to be able to decide when/where to use it. Everything in here is probably (?) old news for people already way into using LLMs to write code.

I decided that a project to speed up my rspec test suite (using the amazing test-prof for profiling and performance patterns!) was a great first application of it — because it will probably involve both analysis and writing many files, if nothing else Claude is probably great at editing many files according to my instructions doing much more than a regex grep can do (yes, indeed it was great at this).

Since I’m optimizing the test suite, I definitely want Claude Code to be able to run rspec — but really for any task, I gather you do, because you definitely want it to be able to run tests to make sure they pass, and iterate if it did something to break tests.

I somewhat unorthodoxly use chruby as my ruby version manager, and I had a bit of trouble getting claude to run rspec (and any other ruby tools I might want) with chruby, and then a bit more trouble when I realized that capybara with selenium-chromedriver was running into trouble with default sandbox that in June 2026 a MacOS Claude Code runs in.

tldr, here’s the PR with the settings/configuration I ended up with.

I was not used to tools that work like Claude, and it all seems to be somewhat under-documented (perhaps because it’s changing so fast) and under-blogged about (do people blog anymore when they can just ask an LLM to solve it so nobody is reading blogs?), or just confusing to me — it took me a day or two to figure it out honestly, and I kept wondering if I was doing it wrong/different from anyone else… but I think what I ended up with is reasonable? If you know better/different, please do let me know!

I definitely kept thinking “surely I’m not the only one trying to do this, why is this so confusing to me and why are others so confused when i ask about? Am I missing something obvious?” I’m still not sure! But I share what I figured out in case it will help.

Specify to run with chruby-exec in a CLAUDE.md

I could not get Claude to run the normal source files for chruby — editing various .bash or .zsh config files (yes I know about ~/.zshenv) did not seem to have any effect. Perhaps Claude Code doesn’t use a ‘real’ shell that uses any config files? When I asked Claude Code itself what to do, it suggested configuration to try to get Claude Code to use config files.. but none seemed to work?

One thing Claude kept suggesting was hard-coding the ENV variables set by chruby in the claude settings.json — which I’m sure would have worked, but I just didn’t like it as a solution. Maybe this is what everyone else is doing? I thought surely we can do better. Plus I’d ideally like it to auto choose based on .ruby-version, not be something I have to update everytime I update ruby (frequent), or have Claude accidentally using a different ruby than my other tools are!

Thanks to @havenwood for helping me think through it on chruby github discussions, and for suggesting using chruby-exec, with a little shell substitution with cat .ruby-version. This in a CLAUDE.md (rather than other things in settings.json) seems to work great:

Prefix Ruby shell commands (ruby, bundle, rake, etc.) with chruby-exec $(cat .ruby-version) --:

“`

chruby-exec $(cat .ruby-version) — bundle exec rake

“`

Running chrome does not work in sandbox used on MacOS

I can’t speak for other OS’s, and I don’t totally understand what’s going on (MacOS “Seatbelt” I guess?), but Claude Code executed rspec was refusing to bring up headless chrome,which i use for system/feature specs via selenium/selenium-webdrivers.

Trying multiple things Claude suggested to specifically allow-list chrome(driver) through the sandbox, definitely none of them worked. Btw, did try switching to cuprite (with Claude Code’s help of course to do it fairly quickly) — despite some reddit suggestions, it seemed to still have the exact same sandbox issue, and at least in my project actually ran my test suite somewhat slower than chromedriver.

Eventually, with more confusing reddit discussion where nobody else had any idea what I was talking about or why I was having a problem, I decided that everyone else must just be exempting rspec itself from the sandbox. (Because surely having Claude Code run rspec is very normal, right? It’s just so useful!) (Thank you to redditors who tried to help!)

It’s just straight rspec, but rspec with many possible arguments, running just certain files/examples, possibly with profiling arguments for test-prof etc. I need it exempted from the sandbox so it can run chrome(driver), but I also need it not to be asking me “Is it okay to run this set of argumetns with rspec” all the time?

Two different settings in Claude’s settings.json, both accept wildcards — for both I want to apply to rspec executions but not accidentally extra stuff, want to try to stay secure-ish here. The chruby rigamorole above makes that somewhat more confusing.

I forget if it was my idea or claude’s idea, but we wrote a wrapper script for chruby-exec-rspec, so we could more cleanly allow-list just that. Claude definitely wrote the implementation of the bash wrapper script. And when I realized that all the test-prof inline ENV vars for profiling (like FPROF=1) messed up my attempted left-anchored allow-listing, I asked claude to work that out by letting the wrapper rearrange an arg into an inline ENV prefix, something my bash skills were def not up to.

The best way to see how I did all that is just the PR.

A word on Claude Desktop

I initially started work in Claude Desktop “code” tab, rather than the CLI. I think this actually made it more confusing to solve these problems above? I am not sure if sandboxing works differnetly in Claude Desktop vs claude CLI? I think the desktop may just be running the claude CLI in various directories?

Just starting out and not being sure how things were working… I found trying to ask Claude [how/] to fix the problems I was having, made things very confusing. Claude does not know whether it’s running as Claude Desktop or not, and was not really sure if the answer is different ha (Claude Desktop probably post-dates Claude Sonnet 4.6’s knowledge base?). Most blogs etc you find googling also pre-date Claude Desktop.

I switched to claude CLI and I can’t totally explain why but things seemed to get simpler. All the fixes I figured out worked when I switched back to Claude Desktop.

And contrary to what you might find googling, claude CLI and Claude Desktop do share sessions now, you can start a session in either place, then move to the other tool to continue it, in either direction. To start claude CLI and choose an existing sessiont to resume, you need to launch as claude --resume.

The CLI is a very neat UI actually! It definitely still seems to be the most popular way to use Claude (whether direct or in a panel in an editor), Claude Desktop “code” tab is I guess fairly new and not as popular, although I like it too and still am mostly using it.

How it worked for the task?

Pretty amazingly actually. Even having read about what it could do, I was kind of amazed.

Once you get it able to run rspec (including with test-prof profiling), this prompt is pretty amazing and fun:

Please use various test-prof profiling commands to identify current best opportunities for speeding up test suite.

Come back ~20 minutes later (my full test suite took ~4 minutes to run at the beginning) and it had some stuff. It tended to just go ahead and make the changes not outline them to me first (I was not in “plan” mode, haven’t tried that much yet), but I’m in a git-controlled dir I can git diff to see what it did — and ask it about it.

By the time I thought to use this general one, i had already implemented some low-hanging optimizations, so that may be why, to be fair, this prompt alone didn’t find much actually significant at that point, honestly.

Here are some others that were pretty amazing:

I am looking to speed up the test suite in this Rails app using rspec and factorybot.

To begin with, let’s focus on the system specs in spec/system. I don’t think they have any obvious performance improvement opportunities. But I’m wondering if they are all necessary. Can you identify any that may be testing something that is not necessary to test, or could be tested by a different kind of spec that is faster?

for specs in spec/components, let’s try changing factory data from create to build_stubbed. Change it for setup where tests still pass. For tests that break when you do that, list them, and if it’s clear let me know why they failed. Analyze performance gains.

[didn’t actually get any gains there, but found that out quickly with very little manual effort, which is a win!]

in our rspec setup, switch from chromedriver to cuprite. make sure tests still pass, if not identify why not.

using AnyFixture, I’ve created a :standard_work fixture, that’s just a generic public tiff-based work.

Can you identify model or service specs it would work well for?

[Didn’t actually end up using AnyFixture yet, but Claude Code helped me make that decision much quicker than I could have without it, based on how much benefit we got vs complexity]

My Rails app uses rspec and Factorybot.

There is an :asset factory with an :inline_promoted_file trait. it turns out this is really bad for performance, and we in fact rarely need to actually create assets with inline promoted files. That should only be used in cases where we really need to test end-to-end derivative and characterization.

In most other cases, we can use a faster “faked file” approach instead. Instead of a trait, we’ve implemented this with a sub-factory, :asset_with_faked_file.

Can you find uses of the :asset factory with :inline_promoted_file, and, if there’s no reason they need to test end-to-end derivative creation, change them to use :asset_with_faked_file sub-factory instead?

[It was able to identify the ones that would work pretty well, was the amazing part — and explain to me exactly why the other ones wouldn’t]

Tell me if I’m doing something weird?

Some of the stuff with chruby/rspec, I am still surprised I had so much trouble getting started, and am wondering if I’m doing something weird/wrong!

But I think probably it’s just that I have been writing code so long, that dealing with these tools that work very differently requires my brain to get out of it’s rut… also that I’m kind of a perfectionist and want to understand whats’ going on and be comfortable with it and that it’s the best way, when increasingly others are just vibing? I don’t know!

But feedback welcome!

ActiveRecord neighbor vector search, with per-document max

I am doing LLM “RAG” with rails ActiveRecord, postgres with the pgvector extension for vector similarity searches, and the neighbor gem. I am fairly new to all of this stuff, figuring it out by doing it.

I realized that for a particular use, I wanted to get some document diversity — so i wanted to do a search of my chunks ranked by embedding vector similarity, getting the top k (say 12) chunks — but in some cases I only want, say, 2 chunks per document. So the top 12 chunks by vector similarity, such that only 2 chunks per interview max are represented in those 12 top chunks.

I decided I wanted to do this purely in SQL, hey, I’m using pgvector, wouldn’t it be most efficient to have pg do the 2-per-document limit?

  • Note: This may be a use case that isn’t a good idea! I have come to realize that maybe I want to just fetch 12*3 or *4 docs into ruby, and apply my “only 2 per document” limit there? Because I may want to do other things there anyway that I can’t do in postgres, like apply a cross-model re-ranker? So I dunno, but for now I did it anyway.

So this was some fancy SQL, i was having trouble figuring out how to do it myself, so I asked ChatGPT, sure. It gave me an initial answer that worked, but…

  • Turns out was over-complicated, a simpler (to my understanding anyway) approach was possible
  • Turns out was not performant, it was not using my postgres ‘HNSW’ indexes to make vector searches higher performance, and/or was insisting on sorting the entire table first defeating the point of the indexes. How’d I know? Well, I noticed it was being slower than expected (several seconds or at times much more to return), and then I did postgres explain/analyze… which I had trouble understanding… so i fed the results to ChatGPT and/or Claude, who confirmed, yeah buddy, this is a bad query, it’s not using your vector index properly.

I had to go on a few back and forths with both ChatGPT and Claude (this is just talking to them in a GUI, not actually using Claude Code or whatever), to get to a pattern that did use my index effectively. They kept suggesting things to me that either just didn’t work, or didn’t actually use the index, etc. I had to actually understand what they were suggesting, and tweak it myself, and have a dialog with them…

But i eventually got to this cool method that can take an arbitrary ActiveRecord relation which already has had neighbor nearest_neighbors query applied to it… and wraps it in a larger query using CTE’s that can limit the results to max-per-document.

I wondered if I should try to share this somewhere (would neighbor gem want a PR?), except… I’m realizing like I said above maybe this is not actually a very useful use case, better to do it in ruby… I’m still not necessariliy getting the performance I expected either, although the analyze/explain says the indexes should be used properly.

So I just share here. Note the original base_relation may be it’s own internal joins to enforce additional conditions on retrieval etc. Assuming each Chunk ActiveRecord model has a document_id attribute which we are using to group for max-per-document. 

# We need to take base_scope and use it as a Postgres CTE (Common Table Expression)
    # to select from, but adding on a ROW_NUMBER window function, that let's us limit
    # to top max_per_interview
    #
    # Kinda tricky, especially to do with good index usage. Got solution from google and talking
    # to LLMs, including having them look at pg explain/analyze output.
    #
    # @param base_relation [ActiveRecord::Relation] original relation, it can have joins and conditions.
    #   It MUST have already had vector distance ordering applied to it with `neighbor` gem.
    #
    # @param max_per_interview [Integer] maximum results to include per interview (oral_history_content_id)
    #
    # @param inner_limit [Integer] how many to OVER-FETCH in inner limit, to have enough even after
    #    applying max-per-interview.
    #
    # @return [ActiveRecord::Relation] that's been in a query to enforce max_per_interview limits. It does
    #   not have an overall limit set, caller should do that if desired, otherwise will be effectively
    #   limited by inner_limit.
    def wrap_relation_for_max_per_interview(base_relation:, max_per_interview:, inner_limit:)
      # In the inner CTE, have to fetch oversampled, so we can wind up with
      # hopefully enough in outer. Leaving inner unlimited would be peformance problem,
      # cause of how indexing works it doesn't need to calculate them all if limited.
      base_relation = base_relation.limit(inner_limit)

      # Now we have another CTE that assigns doc_rank within partitioned
      # interviews, from base. Raw SQL is just way easier here.
      partitoned_ranked_cte = Arel.sql(<<~SQL.squish)
        SELECT base.*,
          ROW_NUMBER() OVER (
            PARTITION BY document_id
            ORDER BY neighbor_distance
          ) AS doc_rank
        FROM base
      SQL

      # A wrapper SQL that incorporates both those CTE's, limiting to
      # doc_rank of how many we want per-interview, and overall making sure to
      # again order by vector neighbor_distance that must already have been included
      # in the base relation.
      base_relation.klass
        .select("*") # just pass through from underlying CTE queries.
        .with(base: base_relation)
        .with(partitioned_ranked: partitoned_ranked_cte)
        .from("partitioned_ranked")
        .where("doc_rank <= ?", max_per_document)
        .order(Arel.sql("neighbor_distance"))
    end

Like I said, I am new to this LLM stuff, curious what others have to say here.

Help fund attorney for artist charged with transporting zines(?!?)

i know Des Revol, and know them to be an incredibly kind, solid, reliable person.

For real he’s facing federal charges and threat of deportation because of subversive political pamphlets found in his trunk.

Des was not at the Prairieland demonstration. Instead, on July 6, after receiving a phone call from his wife in jail (one of the initial ten), Des was followed by Federal Bureau of Investigation (“FBI”) agents in Denton, Texas. They pretextually pulled him over due to a minor traffic violation and quickly arrested him at gunpoint. He was later charged with alleged “evidence tampering and obstruction of justice” based on a box of political pamphlets that he purportedly moved in his truck from his home (not his wife’s) to another house. This type of literature can be found in any activist house or independent bookstore. Des was briefly held at the Johnson County Jail, and then transferred to a federal prison, FMC Fort Worth, where he has been held ever since.

He is also currently on an ICE hold, and has been publicly targeted and doxxed on social media by both prominent fascists and ICE. Moreover, right after his arrest, his family experienced a brutal and intimidating nine-hour FBI raid of their home. Police confiscated everything from electronics to stickers and more zines.

I’m a librarian (and software engineer, but I have a librarian’s MLIS degree and have made a career in libraries). I know that if collecting and distributing controversial, dissident, and even “subversive” political literature is subject to this kind of state repression, our entire society is in trouble.

Attorneys are expensive. And they are all so busy right now.

If you can spare a few bucks, care about a free society, and feel that supporting Des is a good way to do it, please help contribute at his GoFundMe.

More info in this article from the Intercept, and at Des’ support website.

Des says:

I want to be very clear. I did not participate. I was not aware nor did I have any knowledge about the events that transpired on July 4 outside the Prairieland Detention Center. Despite not having any knowledge or not having been near the area at all, I was violently arrested at gunpoint for allegedly making a “wide turn.” My feeling is that I was only arrested because I’m married to Mari Rueda, who is being accused of being at the noise demo showing support to migrants who are facing deportation under deplorable conditions. For this accusation, she’s being threatened with a life sentence in prison.

My charge is allegedly having a box containing magazine “zines,” books, and artwork. Items that are in the possession of millions of people in the United States. Items that are available free online, and available to purchase at stores and online even at places like Amazon. Items that should be protected under the First Amendment “freedom of speech.” If this is happening to me now, it’s only a matter of time before it happens to you.

I believe there’s been almost 20 people arrested in supposed relation to this public noise demo. More than half of those were arrested days later despite not being in the area and are now facing a slew of outrageous charges, in what seems like a political persecution to instill fear on people exercising their First Amendment right.

Whisper-generated transcripts used in presentation of archival video

Here at the Science History Institute, we have a fairly small, but growing, body of video/film in our Digital Collections, at present just over 100 items, around 70 hours total.

We wanted to add transcripts/captions to these videos, for accessibility to those who are hearing impaired, for searchability of video transcript content, and for general usability. We do not have the resources to do any manual transcription or even really Quality Assurance, but we decided that OpenAI whisper automated transcription software was of sufficient quality to be useful.

We have implemented whisper-produced transcriptions. We use them for on-screen text track captions; for an accompanying on-the-side transcript; and for indexing for searching in our collection.

I’ll talk about some of the choices we made and things we discovered, including: our experience using whisper to transcribe; implementing a text track for captions in the video screen (and some Safari weirdness with untitled empty track); synchronized transcript elsewhere on the page; improving the default video.js skin/theme; and trying to encourage Google to index transcript text.

Baseline: The Chemist, an amusing 1970s kind of impressionistic/conceptual promotional video for… chemists being really cool?

Some other interesting videos in our collection

OpenAI Whisper Hosted API

Many of our library/museum/archives peers use the open source Whisper implementation, or a fork/variation of it, and at first I assumed I would do the same. However, we deploy largely on heroku, and I quickly determined that the RAM requirements (at least for medium and above models, and disk space requirements (a pip install openai-whisper added tens of gigs) were somewhere in between inconvenient and infeasible on the heroku cedar platform, at least for our budget.

These limitations and costs change on the new heroku fir platform, so at first I thought we might have to wait until we migrate there — but then I noticed whisper also existed, of course, on the commercial OpenAI API platform.

This is not exactly the same product as OpenAI whisper, and exactly how it differs is not public. The hosted whisper does not let (or require?) you to choose a model, it just uses whatever it uses. It has fewer options — and in the open source realm, there are forks or techniques with even more options and features, like diarization or attempting to segment multi-lingual recordings by language. With the hosted commercial implementation, you just get what you get.

But on the plus side, it’s of course convenient not to have to provison your own resources. It is priced at $0.006 per minute of source audio, so that’s only around $25 to transcribe our meager 70 hour corpus, no problem, and no problem if we keep adding 70-200 hours of video a year as currently anticipated. If we start adding substantially more, we can reconsider our implementation.

Details of whisper API usage implementation

Whisper hosted API has a maximum filesize of 25 MB. Some of our material is up to two hours in length, and audio tracks simply extracted from this material routinely exceeded this limit. But by using ffmpeg to transcode to the opus encoding in an ogg container, using the opus voip profile optimized for voice, at a 16k bitrate — even 2 hours of video is comfortably under 25MB. This particular encoding was found often recommended on forums, with reports that downsampling audio like this can even result in better whisper results; we did not experiment, but it did seem to perform adequately.

ffmpeg -nostdin -y -i input_video.mp4 -vn -map-metadata -1 -ac 1 -c:a libopus -b:a 16k -application voip ./output.oga

Whisper can take a single source language argument — we have metadata already in our system recording language of source material, so if there is only one listed, we supply that. Whisper can’t really handle multi-lingual content. Almost all of our current video corpus is only English, but we do have one video that is mixed English and Korean, and fairly poor audio quality — whisper API actually refused to transcribe that, actually returning an error message (after a wait). When I tried that with opensource whisper just out of curiosity, it did transcribe it, very slowly — but all the Korean passages were transcribed as “hallucinated” English. So error-ing out may actually be a favor to us.

You can give whisper a “prompt” — it’s not conversational instructions, but is perhaps treated more like a glossary of words used. We currently give it our existing metadata “description” field, and that resulted in successful transcription of a word that never caught on, “zeugmatography” (inventor of MRI initially called it that), as well as correct spelling of “Eleuthère Irénée”. If it’s really just a glossary, we might do even better by taking all metadata fields, and just listing unique words once per word (or even trying to focus on less common words). But for now description as-is works well.

Here’s our ruby implementation, pretty simple, using the ruby-openai gem for convenience.

  • I had at one point wanted to stream my audio, stored on S3, directly to a HTTP POST to API, without having to download the whole thing to a local temporary copy first. But ruby’s lack of a clear contract/API/shape of a “stream” object strikes again, making interoperability painful. This fairly simple incompat was just the first of many I encountered; patching this one locally just let me onto the next one, etc. One of my biggest annoyances in ruby honestly!

Results?

As others have found, the results of whisper are quite good, better than any other automated tool our staff had experimented with, and we think the benefits to research and accessibility remain despite what errors do eist. There isn’t much to say about all the things it gets right, by listing the things it doesn’t you might get the wrong idea, but it really does work quite well.

  • As mentioned, it can’t really handle multi-lingual texts
  • Errors and hallucinations were certainly noticed. In one case it accurately transcribed a musical passage as simply ♪, but oddly labelled it as “Dance of the Sugar Plum Fairies” (it was not). An audience clapping was transcribed as repeated utterances of “ok”. This example might be more troubling: some totally imaginary dialog replacing what is pretty unintelligible dialog in the original.
  • Perhaps the most troubling noticed is invented copyright attributions, such as © transcript Emily Beynon (apparently a common one?) — and some other names too. Putting imaginary erroneous copyright declarations in is not great. I am contemplating post-processing to strip any cue beginning with ©, which I think can’t possibly be legitimate?
  • Wide differences in how long the cues are, although consistent within a piece. But some pieces are transcribed with long paragraph-sized cues, and others just phrase by phrase. I am considering post-processing to join tiny phrase cues into sentences, up to so many words.
  • It seems to not infrequently, well into a video, start losing the synchronization of timing, getting 5, 10, or even 15 seconds behind? This is weird and I haven’t seen it commented upon before. The text is still as correct as ever, so mostly an inconvenience. See for instance at 9:09 in Baseline: The Chemist, definitely annoying. By 10:23 it’s caught up again, but quickly gets behind again, etc.

We don’t really have the resources to QA even our fairly small collection, so we are choosing to follow in the footsteps of WGBH and their American Archive of Public Broadcasting, and publish it anyway, with a warning influenced by theirs:

I think in the post-pandemic zoom world, most users are used to automatically generated captions and all their errors, and understand the deal.

WGBH digitizes around 37K items a year, far more than we do. They also run an instance of FixIt+ for public-contributed “crowd-sourced” transcription corrections. While I believe FixIt+ is open source (or a really old version of it is?) and some other institutions may run it, we don’t think we’d get enough public attention and only have a small number of videos, we can’t really afford to stand up our own FixitPlus even if it is available. But it does seem like there is an unfilled need for someone to run a crowd-hosted FixitPlus to charge a reasonable rate for hosting for someone that only will need a handful a year?

We did implement an admin feature to allow upload of corrected WebVTT, which will be used in preference to the direct ASR (Automated Speech Recognition) ones. As we don’t anticipate this being done in bulk, right now staff just downloads the ASR WebVTT, uses the software of their choice to edit it, and then uploads a corrected version. This can be done for egregious errors as noticed, or using whatever policy/workflow our archival team thinks appropriate. We also have an admin feature to disable transcoding for material it does not work well for, such as multi-lingual, silent, or other problems.

Text Track Captions on Video

We were already using video.js for our video display. It provides API’s based on HTML5 video API’s, in some cases polyfilling/ponyfilling, in some cases just delegating to underlying APIs. It has good support for text tracks. At present, by default it uses ‘native’ text tracks instead of it’s own implementation (maybe only on?) Safari — you can force emulated text tracks, but it seemed advisable to stick to default native. This does mean it’s important to test on multiple browsers, there were some differences in Safari that required workarounds (more below).

So, for text tracks we simply provide a WebVTT file in a <track> element under the <video> element. Auto-generated captions (ASR, or “Automated Speech Recognition”, compare to OCR), don’t quite fit the existing categories of “captions” vs “subtitles” — we label them as kind captions and give them an English label “Auto-captions”, which we think/hope is a common short name for these.

Safari adding extra “Untitled” track for untagged HLS

For those most part, this just works, but there was one idiosyncracy that took me a while to diagnose and determine appropriate fix. We deliver our video as HLS with a .M3U8 playlist. There is a newer metadata element in .m3u8 playlist that can label the presence or absence of subtitles embedded in the HLS. But in the absence of this metadata — Safari (both MacOS and iOS I believe) insists on adding a text caption track called “Untitled”, which in our case will be blank. This has been noticed by some, but not as much discussion on the internet as I’d expect to be honest!

One solution would be adding the metadata saying no text track is present embedded in HLS (since we want to deliver text tracks as external in <track> element instead). Somewhat astoundingly, simply embedding an EXT-X-MEDIA tag with a fixed static value of CLOSED-CAPTIONS=NONE — on AWS Elemental MediaConvert (which I use) seems to takes you into the “Professional Tier” costing 60% more! I suppose you could manually post-process the .m3u8 manifests yourself… including my existing ones…

Instead, our solution is simply, when on Safari, hook into events on video element to remove a text track with empty string language and title, which is what characterizes these. I adapted from similar solution in ramp, who chose this direction. They wrote theirs to apply to “mobile which is not android”; I found it actually was needed on Safari (iOS or MacOS Safari too), and indeed not Android Chrome (or iOS Chrome!).

I lost at least a few days figuring out what was going on here and how to fix it, hopefully you, dear reader, won’t have to!

Synchronized Transcript on page next to video

In addition to the text track caption in the video player, I wanted to display a synchronized transcript on the page next to/near the video. It should let you scroll through the transcript independent of the video, and click on a timestamp to jump there.

Unsure of how best to fit this on the screen with what UX — I decided to look at YouTube and base my design on what they did. (On YouTube, you need to expand description and look for a “show transcript” button at bottom of it — I did make my ‘show transcript’ button easier to find!)

It shows up next the video, or when on a narrow screen right below it. In a ‘window in window’ internal scrolling box. Used some CSS to try to make the video and the transcript fit wholly on the screen at any screen size — inner scrolling window that’s higher than the parent window I consider a UX nightmare to avoid!

Looking at YouTube, I realized that feature that highlighted current cue as the video played was also one I wanted to copy. That was the trickiest thing to implement.

I ended up using the HTML5 media element api and the events emitted by it and associated child objects, based on the text track with cues I had already loaded in my video.js-enhanced html5 video player. I can let the browser track cue changes and listen for events when they change, to highlight current cue.

  • If a track is set to mode hidden, then the user agent will still track the text cues and emit events for when they change, even though they aren’t displayed. Video.js (and probably native players) by default have UI that toggles between shown and disabled (which does not track cue changes), so I had to write a bit of custom code to switch non-selected text tracks to hidden instead of disabled
    • (Some browsers and/or video.js polyfill code may have been emitting cueChange events even on disabled tracks, contrary to or not required by spec — important to test on all browsers!)
  • After that, it’s just listening to the cueChange HTML5 video event emitted on the track of our auto-captions, to know that we need to de-highlight any old cues, and highlight the new ones.
  • Had to write code to map from the HTML5 video Cue object returned as active cue, and find the div/span on page to highlight. as simple as putting start time in a data- attribute, and matching it to startTime on Cue — except we’re string-matching, so important to output identically including digits after decimal place etc.
  • At first I didn’t realize I could use the user-agent’s own cue-tracking code, and was trying to catch an event on every timeUpdate event, and calculate which cues included that timestamp myself. In addition to being way more work than required (the HTML5 video API has this feature for you to use) — safari wasn’t emitting timeUpdate events unless the status bar with current time was actually on screen!
  • In general, the media element api and events seemed to an area with, for 2025, unusual level of differences between browsers — or at least between more native Safari and more emulated video.js in other browsers. It definitely is important to do lots of cross browser testing. While I use it rarely, when I do I couldn’t do without BrowserStack and its free offerings for open source.

Improved Video Controls

The default video.js control bar seems to me undesirably small buttons and text, and just not quite right in several ways. And there don’t seem to be very many alternative open source theme or skins (video.js seems to use both words for this), and what do exist are often kind of pushing on “interesting” aesthetics instead of being neutral/universal?

Adding the caption button was squeezing the default control bar tight, especially on small screens. With that and the increased attention to our videos that transcripts would bring, we decided to generally improve the UX of the controls, but in a neutral way that was still generic and non-branded. Again, I was guided by both youtube and the ramp player (here’s one ramp example), and also helped by ramp’s implementation (although beware some skin/theme elements are dispersed in other CSS too, not all in this file).

Before (default video.js theme)

After (local tweaked)

  • Scrubber/progress bar extends all the way across the the screen, above the control bar (ala youtube and ramp)
    • Making sure captions stay above the now higher controls was tricky. I think this approaching using translateY works pretty well, but hadn’t seen it before? Also required a bit of safari-specific css for safari’s “native text tracks”. And some nice slide up/down animation on control bar show/hide matching youtube seems nice.
    • buttons split between right and left, like again both youtube and ramp. Volume on right only cause it was somewhat easier.
  • Buttons themselves made bigger by default, and the icons on the buttons take up a larger portion of the button square. (They were all so tiny before!)
  • Underline the CC button when a text track is visible. From both youtube and ramp.
  • Current time showing as current / total instead of by default elapsed, now matching youtube and what some of our users asked for. (Default video.js has some weird spacing that you have to really trim down once you show current and total).
  • Use newer CSS @container queries to make buttons smaller and/or remove some buttons when screen is smaller (had some weird problems with this actually crashing the video player in my actual markup though).

While fairly minor changes, I think it results in much better look and usability for a general purpose neutral theme/skin than video.js ships with out of the box. While relatively simple, it still took me a week or so to work through.

If there’s interest, I would find time to polish it up further and release it as more easily re-usable open source product, let me know?

Google indexable transcripts

One of the most exciting things about adding transcripts for our videos, is that text is now searchable and discoverable in our own web app.

It would be awfully nice if Google would index it too, so people could find otherwise hidden mentions of things they might want in videos. In the past, I’ve had trouble getting Google to index other kinds of transcripts and item text like OCRs. While hypothetically Google is visiting with javascript and can click on things like tabs or disclosure “show” buttons — conventional wisdom seems to be that Google is doens’t like to index things that aren’t on the initial page and require a click to see — which matches my experience, although others have had other experiences.

In an attempt to see if I could get google to index, I made a separate page with just transcript text — it links back to the main item page (with video player), and even offers clickable timecodes that will link back to player at that time. This transcript-only page is the href on the “Show Transcript” button, although a normal human user ordinarily would get JS executing to show transcript on same page instead when clicking on that link, you can right-click “open in new tab” to get it if you want. These extra transcript pages are also listed in my SiteMap.

There are already a few of these transcript pages showing up in google, so it seems to be a potentially useful move.

That isn’t to say how much SEO juice they have; but first step is getting them in the index, which I had trouble doing before with things that required a tab or ‘show’ click to be shown. So we’ll keep an eye on it! Of course, another option is making the transcript on-page right from the start without requiring a click to show, but I’m not sure if that really serves the user?

We also marked up our item pages with schema.org content for video, including tags around the transcript text (which is initially in DOM, but requires a ‘show transcript’ click to be visible). I honestly would not expect this to do much for increasing indexing of transcripts… I think according to google this is intended to give you a “rich snippet” for video (but not to change indexing)… but some people think Google doesn’t do too much of that anyway, and to have any chance I’d probably have to provide a persistent link to video as a contentUrl which I don’t really do. Or maybe it could make my content show up in Google “Video” tab results… but no luck there yet either. Honestly I don’t think this is going to do much of anything, but it shouldn’t hurt.

Acknowledgements

Thanks to colleagues in Code4Lib and Samvera community slack chats, for sharing their prior experiences with whisper and with video transcripts — and releasing open source code that can be used as a reference — so I didn’t have to spend my time rediscovering what they already had!

Especially generous were Mason Ballengee and Dananji Withana who work on the ramp project. And much thanks to Ryan “Harpo” Harbert for two sequential years of Code4Lib conference presentations on whisper use at WGBH (2024 video, 2025 video), and also Emily Lynema for a 2025 whisper talk.

I hope I have helped pass on a portion of their generosity by trying to share all this stuff above to keep others from having to re-discover it!

Using CloudFlare Turnstile to protect certain pages on a Rails app

I work at a non-profit academic institution, on a site that manages, searches, and displays digitized historical materials: The Science History Institute Digital Collections.

Much of our stuff is public domain, and regardless we put this stuff on the web to be seen and used and shared. (Within the limits of copyright law and fair use; we are not the copyright holders of most of it). We have no general problem with people scraping our pages.

The problem is that, like many of us, our site is being overwhelmed with poorly behaved bots. Lately one of the biggest problems is with bots clicking on every possible combination of facet limits in our “faceted search” — this is not useful for them, and it overwhelms our site. “Search” pages are one of our most resource-constrained category of page in our present site, adding to the injury. Peers say even if we scaled up (auto or not) — the bots sometimes scale up to match anyway!

One option would be putting some kind of “Web Application Firewall” (WAF) in front of the whole app. Our particular combination of team and budget and platform (heroku) makes a lot of these options expensive for us in licensing, staff time to manage, or both. Another option is certainly putting the the whole thing behind (ostensibly free) CloudFlare CDN and using its built-in WAF, but we’d like to avoid giving our DNS over to CloudFlare, I’ve heard mixed reviews of CloudFlare free staying free, and generally am trying to avoid contributing to CloudFlare’s monopoly unaccountable control of the internet.`

Although ironically then, the solution we arrived at is still using CloudFlare, but Cloudflare’s Turnstile “captcha replacement”, one of those things that gives you the “check this box” or more often entirely interactive “checking if you are a bot” UXs.

[If you’re a tldr look at the code type, here’s the initial implementation PR in our open repo, there are some bug fixes since then
Update March 18 2025: There is now a gem implementation, bot_challenge_page. It is pre-1.0 and still evolving as we learn more about the problem space]

While this still might unfortunately lock people using unconventional browsers etc out (just the latest of many complaints on HackerNews), we can use this to only protect our search pages. Most of our traffic comes directly from Google to an individual item detail page, which we can now leave completely out of it. We have complete control of allow-listing traffic based on whatever characteristics, when to present the challenge, etc. And it turns out we had a peer at another institution who had taken this approach and found it successful, so that was encouraging.

How it works: Overview

While typical documented Turnstile usage involves protecting form submissions, we actually want to protect certain urls, even when accessed via GET. Would this actually work well? What’s the best way to implement it?

Fortunately, when asking around on a chat for my professional community of librarian and archivist software hackers, Joe Corall from Lehigh University said they had done the exact same thing (even in response to the same problem, bots combinatorially exploring every possible facet value), and had super usefully written it up, and it had been working well for them.

Joe’s article and the flowchart it contains is worth looking it. His implementation is as a Drupal plugin (and used in at least several Islandora instances); the VuFind library discovery layer recently implemented a similar approach. We have a Rails app, so needed to implement it ourselves — but with Joe paving the way (and patiently answering our questions, so we could start with the parameters that worked for him), it was pretty quick work, bouyed by the confidence this approach wasn’t just an experiment in the blue, but had worked for a similar peer.

  • Meter the rate of access, either per IP address, or as Joe did, in buckets per sub-net of client IP address.
  • Once client has crossed a rate limit boundary (in Joe’s case 20 requests per 24 hour period), redirect them to a page which displays the Turnstile challenge — and has the original destination in a query param in url —
  • Once they have passed the Turnstile challenge, redirect them back to their original destination, which now lets them in because you’ve stored their challenge pass in some secure session state.
  • In that session state record that they passed, and let them avoid a challenge again for a set period of time.

Joe allow-listed certain client domain names based on reverse IP lookup, but I’ve started without that, not wanting the performance hit on every request if I can avoid it. Joe also allow-listed their “on campus” IPs, but we are not a university and only have a few staff “on campus” and I always prefer to show the staff the same thing our users are seeing — if it’s inconvenient and intolerable, we want to feel the pain so we fix it, instead of never even seeing the pain and not knowing our users are getting it!

I’m going to explain and link to how we implemented this in a Rails app, and our choices of parameters for the various parameterized things. But also I’ll tell you we’ve written this in a way that paves the way to extracting to a gem — kept everything consolidated in a small number of files and very parameterized — so if there’s interest let me know. (Code4Lib-ers, our slack is a great place to get in touch, I’m jrochkind).

Ruby and Rails details, and our parameters

Here’s the implementing PR. It is written in such a way to keep the code conslidated for future gem extraction, all in the BotDetectController class, which means kind of weirdly there is some code to inject in class methods in the controller. While it does turnstile now, it’s written with variable/class names such that analagous products could be made available.

Rack-attack to meter

We were already using rack-attack to rate-limit. We added a “track” monitor with our code to decide when a client had passed a rate-limit gate to require a challenge. We start with allowing 10 requests per 12 hours (Joe at Lehigh did 20 per 24 hours), batched together in subnets. (Joe did subnets too, but we do smaller /24 (ie x.y.z.*) for ipv4 instead of Joe’s larger /16 (x.y.*.*)).

Note that rack-attack does not use sliding/rolling-windows for rate limits, but fixed windows that reset after window period. This makes a difference especially when you use such a long period as we are, but it’s not a problem with our very low count per period, and it does keep the RAM extremely effiicent (just an integer count per rate limit bucket).

When the rate limit is reached, the rack-attack block just sets a key/value in the rack_env to tell another component that a challenge is required. (setting in the session may have worked, but we want to be absolutely sure this will work even if client is not storing cookies, and this is really only meant as this-request state, so rack env seemed the good way to set state in rack-attack that could be seen in a rails controller)

Rails before_action filter to enforce challenge

There’s a Rails before_action filter that we just put on the application-wide ApplicationController, that looks for the “bot challenge key” required in the rack env — if present, and there isn’t anything in the session saying they have already passed a bot challenge, then we redirect to a “challenge” page, that will display/activate Turnstile.

We simply put the original/destination URL in a query param on that page. (And include logic to refuse to redirect to anything but a relative path on same host, to avoid any nefarious uses).

The challenge controller

One action in our BotDetectController just displays the turnstile challenge. The cloudflare turnstile callback gives us a token we need to verify server-side with turnstile API to verify challenge was really passed.

the front-end does a JS/xhr/fetch request to the second action in our BotDetectController. The back-end verify action makes the API call to turnstile, and if challenge passed, sets a value in Rails (encrypted and signed, secure) session with time of pass, so the before_action guard can give the user access.

if the JS in front gets a go-ahead from back-end, it uses JS document.replace to go to original destination. This conveniently removes the challenge page from the user’s browser history, as if it never happened, browser back button still working great.

In most cases the challenge page, if non-interactive, wont’ be displayed for more than a few seconds. (the language has been tweaked since these screenshots).

We currently have a ‘pass’ good for 24 hours — once you pass a turnstile challenge, if your cookies/session are intact, you won’t be given another one for 24 hours no matter how much traffic. All of this is easily configurable.

If the challenge DOES fail for some reason, the user may be looking at the Challenge page with one of two kinds of failures, and some additional explanatory text and contact info.

Limitations and omissions

This particular flow only works for GET requests. It could be expanded to work for POST requests (with an invisible JS created/submitted form?), but our initial use case didn’t require it, so for now the filter just logs a warning and fails for POST.

This flow also isn’t going to work for fetch/ajax requests, it’s set up for ordinary navigation, since it redirects to a challenge then redirects back. Our use case is only protecting our search pages — but the blacklight search in our app has a JS fetch for “facet more” behavior. Couldn’t figure out a good/easy way to make this work, so for now we added an exemption config, and just exempt requests to the #facet action that look like they’re coming from fetch. Not bothered that an “attacker” could escape our bot detection for this one action; our main use case is stopping crawlers crawling indiscriminately, and I don’t think it’ll be a problem.

To get through the bot challenge requires a user-agent to have both JS and cookies enabled. JS may have been required before anyway (not sure), but cookies were not. Oh well. Only search pages are protected by the bot challenge.

The Lehigh implementation does a reverse-lookup of the client IP, and allow-lists clients from IP’s that reverse lookup to desirable and well-behaved bots. We don’t do that, in part because I didn’t want the performance hit of the reverse-lookup. We have a Sitemap, and in general, I’m not sure we need bots crawling our search results pages at all… although I’m realizing as I write this that our “Collection” landing pages are included (as they show search results)… may want to exempt them, we’ll see how it goes.

We don’t have any client-based allow-listing… but would consider just exempting any client that has a user-agent admitting it’s a bot, all our problematic behavior has been from clients with user-agents appearing to be regular browsers (but obviously automated ones, if they are being honest).

Possible extensions and enhancements

We could possibly only enable the bot challenge when the site appears “under load”, whether that’s a certain number of overall requests per second, a certain machine load (but any auto-scaling can make that an issue), or size of heroku queue (possibly same).

We could use more sophisticated fingerprinting for rate limit buckets. Instead of IP-address-based, colleague David Cliff from Northeastern University has had success using HTTP user-agent, accept-encoding, and accept-language to fingerprint actors across distributed IPs, writing:

I know several others have had bot waves that have very deep IP address pools, and who fake their user agents, making it hard to ban.

We had been throttling based on the most common denominator (url pattern), but we were looking for something more effective that gave us more resource headroom.

On inspecting the requests in contrast to healthy user traffic we noticed that there were unifying patterns we could use, in the headers.

We made a fingerprint based on them, and after blocking based on that, I haven’t had to do a manual intervention since.

def fingerprint
result = “#{env[“HTTP_ACCEPT”]} | #{env[“HTTP_ACCEPT_ENCODING”]} | #{env[“HTTP_ACCEPT_LANGUAGE”]} | #{env[“HTTP_COOKIE”]}”
Base64.strict_encode64(result)
end

…the common rule we arrived at mixed positive/negative discrimination using the above

request.env["HTTP_ACCEPT"].blank? && request.env["HTTP_ACCEPT_LANGUAGE"].blank? && request.env["HTTP_COOKIE"].blank? && (request.user_agent.blank? || !request.user_agent.downcase.include?("bot".downcase))

so only a bot that left the fields blank and lied with a non-bot user agent would be affected

We could also base rate limit or “discriminators” for rate limit buckets on info we can look up from the client IP address, either a DNS or network lookup (performance worries), or perhaps a local lookup using the free MaxMind databases that also include geocoding and some organizational info.

Does it work?

Too early to say, we just deployed it!

I sometimes get annoyed when people blog like this, but being the writer, I realized that if I wait a month to see how well it’s working to blog — I’ll never blog! I have to write while it’s fresh and still interesting to me.

But encouraged that colleagues say very similar approaches have worked for them. Thanks again to Joe Corral for paving the way with a drupal implementation, blogging it, discussing it on chat, and answering questions! And all the other librarian and cultural heritage technologists sharing knowledge and collaboration on this and many other topics!

I can say that already it is being triggered a lot, by bots that don’t seem to get past it. This includes google bot and Meta-ExternalAgent (which I guess is AI-related; we have no particular use-based objections we are trying to enforce here, just trying to preserve our resources). While Google also has no reason to combinatorially explore every facet combination (and has a sitemap), I’m not sure if I should exempt known resource-considerate bots from the challenge (and whether to do so by trusting user-agent or not; our actual problems have all been with ordinary-browser-appearing user-agents).

Update 27 Jan 2025

Our original config — allowing 10 search results per IP subnet before turnstile challenge — was not enough to keep the bot traffic from overwhelming us. Too many botnets had enough IPs making apparently fewer than 10 requests each.

Lowering that to 2 requests was enough to reduce traffic enough. (Keep in mind that a user should only get one challenge per 24 hours unless IP address changes — although that makes me realize that people using Apple’s “private browsing” feature may get more, hmm).

Pretty obvious on these heroku dashboard graphs where our succesful turnstile config was deployed, right?

I think I would be fine going down to challenge on first search results, since a human user should still only get one per 24 hour period — but since the “success passed” mark in session is tied to IP address (to avoid session replay for bots to avoid the challenge), I am now worried about Apple “private browsing”! In today’s environment with so many similar tests, I wonder if private browsing is causing problems for users and bot protections?

You can see on the graph a huge number of 3xx responses — those are our redirects to challenge page! The redirect to and display of the challenge page seem to be cheap enough that they aren’t causing us a problem even in high volume — which was the intent, nice to see it confirmed at least with current traffic.

We are only protecting our search result page, not our item detail pages (which people often get to directly to google) — this also seems succesful. The real problem was the volume of hits from so many bots trying to combinatorially explore every possible facet limit, which we have now put a stop to.

Accessing capybara-screenshot artifacts on Github CI

We test our Rails app with rspec and capybara.

For local testing, we use the capybara-screenshot plugin which “Automatically save screen shots when a Capybara scenario fails”, even when the tests were running in a headless browser you couldn’t see at all. This can be very helpful in debugging tricky capybara failures, especially ones that are “flaky” and hard to reproduce failure on.

We run all our tests automatically as CI in Github Actions.

I was running into some capybara browser tests that were failing flakily and inconsistently on Github Actions, but I could not manage to reproduce locally at all. What was going on? It would be super helpful to have access to the capybara-screenshots generated on the github actions run.

Is there a way to do it? Yes! Store them as Github Actions “artifacts“. My last two steps of my github workflow .yml look like this, the one that runs rspec, and then the one that saves any capybara-screenshot screenshot artifacts!

        - name: Run tests
          run: |
            bundle exec rspec

        - name: Archive capybara failure screenshots
          uses: actions/upload-artifact@v4
          if: failure()
          with:
            name: dist-without-markdown
            path: tmp/capybara/*.png
            if-no-files-found: ignore

I already had capybara-screenshot set up.

Now, if a capybara test fails, I can look at the screenshot filename reported for that particular failed test in the Github CI log.

And, then down under the “Archive capybara failure screenshots” action, I can find a clickable URL, which when clicked on, downloads a zip file that contains any/all archived capybara screenshots. If there are more than one, I can match filenames to the filename reported in a certain spec failure.

And I confirmed that last step with an if: failure() does not change the failure status of the job — the job is still marked by Github CI as failed, as it should be, but the archiving step still runs to archive the failure artifacts.

Very handy!

Getting rspec/capybara browser console output for failed tests

I am writing some code that does some smoke tests with capybara in a browser of some Javascript code. Frustratingly, it was failing when run in CI on Github Actions, in ways that I could not reproduce locally. (Of course it ended up being a configuration problem on CI, which you’d expect in this case). But this fact especially made me really want to see browser console output — especially errors, for failed tests, so I could get a hint of what was going wrong beyond “Well, the JS code didn’t load”.

I have some memory of being able to configure a setting in some past capybara setup, to make error output in browser console automatically fail a test and output? But I can’t find any evidence of this on the internet, and at least I’m pretty sure there is no way to do that with my current use of selenium-webdrivers and with the headless chrome to run capybara tests.

So I worked out this hacky way to add any browser console output to the failure message on failing tests only. It requires using some “private” rspec API, but this is all I could figure out. I would be curious if anyone has a better way to accomplish this goal.

Note that my goal is a bit different than “make a test fail if there’s error output in browser console”, although I’m potentially interested in that too, here I wanted: for a test that’s already failing, get the browser console output, if any, to show up in failure message. 

# hacky way to inject browser logs into failure message for failed ones
  after(:each) do |example|
    if example.exception
      browser_logs = page.driver.browser.logs.get(:browser).collect { |log| "#{log.level}: #{log.message}" }

      if browser_logs.present?
        # pretty hacky internal way to get browser logs into 
        # existing long-form failure message, when that is
        # stored in exception associated with assertion failure
        new_exception = example.exception.class.new("#{example.exception.message}\n\nBrowser console:\n\n#{browser_logs.join("\n")}\n")
        new_exception.set_backtrace(example.exception.backtrace)

        example.display_exception = new_exception
      end
    end
  end

I think by default, with selenium headless chrome, you should get browser console that only includes error/warn log levels but not info, but if you aren’t getting what you want or want more you need to make a custom Capybara driver with custom loggingPrefs config that may look something like this:

Capybara.javascript_driver = :my_headless_chrome

Capybara.register_driver :my_headless_chrome do |app|
  Capybara::Selenium::Driver.load_selenium
  browser_options = ::Selenium::WebDriver::Chrome::Options.new.tap do |opts|
    opts.args << '--headless'
    opts.args << '--disable-gpu'
    opts.args << '--no-sandbox'
    opts.args << '--window-size=1280,1696'

    opts.add_option('goog:loggingPrefs', browser: 'ALL')
  end
  Capybara::Selenium::Driver.new(app, browser: :chrome, options: browser_options)
end

keyword-like arguments to JS functions using destructuring

I am, unusually for me, spending some time writing some non-trivial Javascript, using ES modules.

In my usual environment of ruby, I have gotten used to really preferring keyword arguments to functions for clarity. More than one positional argument makes me feel bad.

I vaguely remembered there is new-fangled way to exploit modern JS features to do this with JS, including default values, but was having trouble finding it. Found it! It involves using “destructuring”. Putting it here for myself, and in case this text gives someone else (perhaps another rubyist) better hits for their google searches than I was getting!

function freeCar({name = "John", color, model = "Honda"} = {}) {
  console.log(`Hi ${name}, you get a ${color} ${model}`);
}

freeCar({name: "Joe", color: "Green", model: "Lincoln"})
# Hi Joe, you get a Green Lincoln

freeCar({color: "RED"})
# Hi John, you get a RED Honda

freeCar()
# Hi John, you get a undefined Honda

freeCar({})
# Hi John, you get a undefined Honda

Sort order of results in OCLC AssignFAST/fastsuggest API

We have long used the free OCLC AssignFAST API to power an auto-suggest/auto-complete in some of our staff metadata entry forms. (Note: While OCLC calls this service “AssignFAST” in docs, the base URL instead includes the term fastsuggest as in http://fast.oclc.org/searchfast/fastsuggest?, which may be confusing for some!)

Recently our staff reported they thought the sort order had changed in the results returned — before they think they could enter, say, “Philadelphia” and get “Philadelphia–Pennsylvania” as a suggested response — which was the one they wanted. But now, that result wasn’t even in the first 15 results, and thus wasn’t included in our drop-down auto-suggest menu. It looks like the current result order is strictly alphabetical, so includes a lot of obscure or unlikely to be useful hits from a query matching any part of a term.

We contacted OCLC support at this form we found for “Contact the FAST team”. They got back to us relatively quickly (hooray!), but unfortunately could not neither confirm nor deny that any changes had happened to the order of results from that API any time recently, or any time at all.

But they did, eventually, tell us that we could in fact control the sort order of the API response, and make it return results in order of usage, by adding a query param to the API request — &sort=usage+desc. I would guess “usage” means frequency of use in the OCLC catalog record database.

Eg, https://fast.oclc.org/searchfast/fastsuggest?&query=philadelphia&query&queryReturn=suggestall%2Cidroot%2Cauth%2Ctype&suggest=autoSubject&rows=20&sort=usage+desc

This works, and seems to return results more like what we’re expecting/used to?

It also seems to return results more like what their own web page at https://fast.oclc.org/searchfast/ returns?

It’s also entirely undocumented on the AssignFAST API documentation, and they don’t seem inclined to update the docs? I do not know if there might be other useful values selectable as sort fields, in addition to usage.

I make this post in part for Google-ability for anyone else looking to solve this mystery.

The AssignFAST doc page also includes lots of broken links to examples. In general, I think we probably should be grateful this free API still exists at all; I think OCLC is putting only minimal if any resources to even keeping it alive, let alone enhancing it or responding to current user need; I’d expect it to disappear at some point.

I remember a time when I first entered the profession where I imagined we would develop all sorts of innovative digital services and APIs from which a new generation of library technological ecosystem would be built, and OCLC, our community-owned non-profit cooperative, would be at the center of that doing innovative things that they were well-placed to do with their access to metadata ecosystems. Of that fantasized future, what remains in 2024 is the leftover unsupported frozen-on-the-vine initial experiments from that era of excitement and innovation, and a shrunken ecosystem of vendors trying to figure out how to wring every last drop of revenue from their customers.

Cloudfront in front of S3 using response-content-disposition

At the Science History Institute Digital Collections, we have a public collection of digitized historical materials (mostly photographic images of pages). We store these digitized assets — originals as well as various resizes and thumbnails used on our web pages — in AWS S3.

Currently, we provide access to these assets directly from S3. For some of our deliveries, we also use the S3 feature of a response-content-disposition query parameter in a signed expiring S3 url, to have the response include an HTTP Content-Disposition header with a filename and often attachment disposition, so when the end-user saves the file they get a nice humanized filename (instead of our UUID filename on S3), supplied dynamically at download time — while still sending the user directly to S3, avoiding the need for a custom app proxy layer.

While currently we’re sending the user directly to urls in S3 buckets set with public non-authenticated access, we understand a better practice is putting a CDN in front like AWS’s own CloudFront. In addition to the geographic distribution of a CDN, we believe this will give us: better more consistent performance even in the same AWS region; possibly some cost savings (although it’s difficult for me to compare the various different charges over our possibly unusual access patterns); and additionally access to using AWS WAF in front of traffic, which was actually our most immediate motivation.

But can we keep using the response-content-disposition query param feature to dynamically specify a content-disposition header via the URL? It turns out you certainly can keep using response-content-disposition through CloudFront. But we found it a bit confusing to set up, and think through the right combination features and their implications, with not a lot of clear material online.

So I try to document here the basic recipe we have used, as well as discuss considerations and details!

Recipe for CloudFront distribution forwarding response-content-disposition to S3

  • We need CloudFront to forward response-content-disposition header to s3 — by default it leaves off query string (after ? in a URL) when forwarding to origin. You might reach for a custom Origin Request Policy, but it turns out we’re not going to need it, because a Cache Policy will take care of it for us.
  • If we’re returning varying content-disposition headers, we need a non-default Cache Policy such that the cache key varies based on response-content-disposition too — otherwise changing the content-disposition in query param might get you a cached response with old stale content-disposition.
    • We can create a Cache Policy based on the managed CachingOptimized policy, but adding the query params we are interested in.
    • It turns out including URL query params in a Cache Policy automatically leads to them being included in origin requests, so we do NOT need a custom Origin Request Policy. Only a custom Cache Policy that includes response-content-disposition
  • OK, but for the S3 origin to actually pay attention to the response-content-disposition` header, you need to set up a CloudFront Origin Access Control  (OAC) given access to the S3 bucket, and set to “sign requests”. Since S3 only respects this header for signed requests.
    • You don’t actually need to restrict the bucket to only allow requests from CloudFront, but you probably want to make sure all your buckets requests are going through cloudfront?
    • You don’t need to restrict the CloudFront distro to Restrict viewer access, but there may be security implications of setting up response-content-disposition forwarding with non-restircted distro? More discussion below.
    • Some older tutorials you may find use AWS “Origin Access Identity (OAI)” for this, but OAC is the new non-deprecated way, don’t follow those tutorials.
    • Setting this all up has a few steps, and but this CloudFront documentation page leads you through it.

At this point your Cloudfront distribution is working to forward response-content-disposition headers, and return the resultant content-disposition headers in response — Cloudfront by default forwards on all response headers from origin, by default if you haven’t set a distribution behavior “Response headers policy”. Even setting a response headers policy like Managed-CORS-with-preflight-and-SecurityHeadersPolicy (which is what I often need), it seems it forwards on other response headers like content-disposition no problem.

Security Implications of Public Cloudfront with response-content-disposition

An S3 bucket can be set to allow public access, as I’ve done with some buckets with public content. But to use the response-content-disposition or response-content-type query param to construct a URL that dynamically chooses a content-disposition or content-type — you need to use an S3 presigned url (or some other form of auth I guess), even on a public bucket! “These parameters cannot be used with an unsigned (anonymous) request.”

Is this design intentional? If this wasn’t true, anyone could construct a URL to your content that would return a response with their chosen content-type or content-disposition headers. I can think of some general vague hypothetical ways this could be used maliciously, maybe?

But by setting up a CloudFront distribution as above, it is possible to set things up so an unsigned request can do exactly that. http://mydistro.cloudfront.net/content.jpg?response-content-type=application%2Fx-malicious, and it’ll just work without being signed. Is that a potential security vulnerability? I’m not sure, but if so you should not set this up without also setting the distribution to have restricted viewer access and require (eg) signed urls. That will require all urls to the distribution to be signed though, not just the ones with the potentially sensitive params.

What if you want to use public un-signed URLs when they don’t have these sensitive params; but require signed URLs when they do have these params? (As we want the default no-param URLs to be long-cacheable, we don’t want them all to be unique time-limited!)

Since CloudFront “restricted access” is set for the entire distribution/behavior, you’d maybe need to use different distributions both pointed at the same origin (but with different config). Or perhaps different “behaviors” at different prefix paths within the same distribution. Or maybe there is a way to use custom Cloudfront functions or lambdas to implement this, or restrict it in some other way? I don’t know much about that. It is certainly more convoluted to try to set up something like how S3 alone works, where straight URLs are public and persistent, but URLs specifying response headers are signed and expiring.

Other Considerations

You may want to turn on logging for your CloudFront distro. You may want to add tags to make cost analysis easier.

In my buckets, all keys have unique names using UUID or content digests, such that all URLs should be immutable and cacheable forever. I want the actual user-agents making the request o get far-future cache-control headers. I try to set S3 cache-control metadata with far-future expiration. But if some got missed or I change my mind about what these should look like, it is cumbersome (and has some costs) to try to check/reset metadata on many keys. Perhaps I want the CloudFront distro/behavior to force add/overwrite far-future cache-control header itself? I could do that either with a custom response headers policy (might want to start with one of the managed policies, and copy/paste it modifying to add cache-control header), or perhaps a custom origin request policy that added on a S3 response-cache-control query param to ask S3 to return a far-future cache-control header. (You might want to make sure you aren’t telling the user-agent to cache error messages from origin though!)

You may be interested in limiting to a CloudFront price class to control costs.

Terraform example

Terraform files demonstrating what is described here can be found: https://gist.github.com/jrochkind/4edcc8a4a1abf090a771a3e0324f6187

More detailed explanation below.

Detailed Implementation Notes and Examples

Custom Cache Policy

Creating cache polices discussed in AWS docs.

Documentation that Cache Policy results in query params being included in origin requests from documentation on Control origin requests with a policy.

Although the two kinds of policies are separate, they are related. All URL query strings, HTTP headers, and cookies that you include in the cache key (using a cache policy) are automatically included in origin requests. Use the origin request policy to specify the information that you want to include in origin requests, but not include in the cache key. Just like a cache policy, you attach an origin request policy to one or more cache behaviors in a CloudFront distributionz

You set a cache policy for your distribution (or specific behavior) by editing a Behavior here:

I created the Cache Policy with TTL values from “CachingOptimized” managed behavior, and added the query params I was interested in:

Which looks like this in terraform:

 resource "aws_cloudfront_distribution" "example-test2" {
      # etc
      default_cache_behavior {
          cache_policy_id        = "658327ea-f89d-4fab-a63d-7e88639e58f6"
      }
}

resource "aws_cloudfront_cache_policy"  "jrochkind-test-caching-optimized-plus-s3-params" {
  name        = "jrochkind-test-caching-optimized-plus-s3-params"
  comment     = "Based on Managed-CachingOptimized, but also forwarding select S3 query params"
  default_ttl = 86400
  max_ttl     = 31536000
  min_ttl     = 1
  parameters_in_cache_key_and_forwarded_to_origin {
    enable_accept_encoding_brotli = true
    enable_accept_encoding_gzip   = true

    cookies_config {
      cookie_behavior = "none"
    }
    headers_config {
      header_behavior = "none"
    }
    query_strings_config {
      query_string_behavior = "whitelist"
      query_strings {
        items = [
          "response-content-disposition",
          "response-content-type"
        ]
      }
    }
  }
}

Cloudfrong Origin Access Control (OAC) to sign requests to S3

Covered in CloudFront docs Restrict access to an Amazon Simple Storage Service origin, which lead you through it pretty nicely.

While you could leave off the parts that actually restrict access (say allowing public access), and just follow the parts for setting up an OAC to sign requests… you probably also want to restrict access to s3 so only CloudFront has it, not the public?

Relevant terraform follows. (You may want to use templating feature for the json policy, shown in complete example above). 

resource "aws_cloudfront_distribution" "example-test2" {
    # etc
    origin {
        connection_attempts = 3
        connection_timeout  = 1
        domain_name         = aws_s3_bucket.example-test2.bucket_regional_domain_name
        origin_id           = aws_s3_bucket.example-test2.bucket_regional_domain_name
        origin_access_control_id = aws_cloudfront_origin_access_control.example-test2.id
    }
}

resource "aws_s3_bucket_policy" "example-test2" {
    bucket = "example-test2"
    
    policy = jsonencode(
        {
            Id        = "PolicyForCloudFrontPrivateContent"
            Statement = [
                {
                    Action    = "s3:GetObject"
                    Condition = {
                        StringEquals = {
                            "AWS:SourceArn" = aws_cloudfront_distribution.example-test2.arn
                        }
                    }
                    Effect    = "Allow"
                    Principal = {
                        Service = "cloudfront.amazonaws.com"
                    }
                    Resource  = "arn:aws:s3:::example-test2/*"
                    Sid       = "AllowCloudFrontServicePrincipal"
                  },
            ]
            Version   = "2008-10-17"
        }
    )
}

resource "aws_cloudfront_origin_access_control" "example-test2" {
  description                       = "Cloudfront signed s3"
  name                              = "example-test2"
  origin_access_control_origin_type = "s3"
  signing_behavior                  = "always"
  signing_protocol                  = "sigv4"
}

Restrict public access to CloudFront

We want to require signed urls with our CloudFront distro, similar to what would be required with a non-public S3 bucket directly. Be aware that CloudFront uses a different signature algorithm and type of key than s3 and expirations can be further out.

See AWS doc at Serve private content with signed URLs and signed cookies.

  • Create a public/private RSA key pair
    • openssl genrsa -out private_key.pem 2048
    • extrat just public key with openssl rsa -pubout -in private_key.pem -out public_key.pem
    • Upload the public_key.pem to CloudFront “Public Keys”, and keep the private key in a secure place yourself.
  • Create a CloudFront “Key Group”, and select that public key from select menu
  • In the Distribution “Behavior”, select “Restrict Viewer Access”, to a “Trusted Key Group”, and choose the Trusted Key Group you just created.

Now all CloudFront URLs for this distribution/behavior will need to be signed to work, or else you’ll get an error Missing Key-Pair-Id query parameter or cookie value. See Use signed URLs. (you could also use a signed cookie, but that’s not useful to me right now).

You’ll need the private key to sign a URL. Note that CloudFront uses an entirely different key signing algorithm, protocol, and key than s3 signed urls! Shrine’s S3 docs have a good ruby example of using ruby AWS SDK Aws::CloudFront::UrlSigner, which will by default use a “canned” policy. (I’m not sure the default expiration you’ll get without specifing it in the call, as in that example.)

In terraform, the public key, trusted key group, and distribution settings might look like the following, using a “canned” policy that just has a simple expiration. Passing a custom expiration for 7 days in future might look something like this:

signed_url = signer.signed_url(
  "https://mydistro.cloudfront.net/content.jpg?response-content-disposition=etc",
  expires: Time.now.utc.to_i + 7 * 24 * 60 * 60,
)

Terraform for creating restricted cloudfront access as above:

resource "aws_cloudfront_public_key" "example-test2" {
  comment     = "public key used by our app for signing urls"
  encoded_key = file("public_key-example-test2.pem")
  name        = "example-test2"
}

resource "aws_cloudfront_key_group" "example-test2" {
  comment = "key group used by our app for signing urls"
  items   = [aws_cloudfront_public_key.example-test2.id]
  name    = "example-test2"
}

resource "aws_cloudfront_distribution" "example-test2" {
  # etc
  trusted_key_groups = [aws_cloudfront_key_group.example-test2.id]
}

(Warning, with terraform aws provider v5.53.0, to have terraform remove the trusted_key_groups and have the distro be public again, have to leave in trusted_key_groups = [], rather than remove the key entirely. Perhaps that’s part of how terraform works)