chore(recordings): remove hub dependency on recordings ingestion#14418
Merged
chore(recordings): remove hub dependency on recordings ingestion#14418
Conversation
added 7 commits
February 27, 2023 14:01
Hub is a grab bag of depencencies that are not all required for recordings ingestion. To keep the recordings ingestion lean, we remove the hub dependency and use the postgres and kafka client directly. This should increase the availability of the session recordings workload, e.g. it should not go down it Redis or ClickHouse is down.
Contributor
Author
|
I’m sorry for the very long pr, I will happily accept reviews that suggest breaking this up to smaller parts |
fuziontech
added a commit
that referenced
this pull request
Feb 28, 2023
* master: (53 commits) fix: cut the undeterministic snapshots (#14461) feat(hogql): Events table based on hogql (#14315) fix(breakdown): ensure breakdown sort can sort through values of different types (#14459) feat(capture): gracefully catch non-string tokens (#14453) chore(plugin-server): add healthcheck logging for failure (#14455) fix(tests): Safer migrations (#14452) revert: "fix(person-overrides): add constraints to catch race conditions" (#14445) dev(codespaces): update to python3.10 (#14449) chore(deps): Update posthog-js to 1.50.0 (#14448) feat(capture): support LIGHTWEIGHT_CAPTURE_ENDPOINT_ENABLED_TOKENS=* (#14421) chore: upgrade d3 (#14442) fix(person-overrides): add constraints to catch race conditions (#14277) feat(cohorts): Remove postgres calculations for flags (#14272) feat(capture): check token shape before team resolution too (#14439) feat: dashboard templates (#14322) feat: add a 'What's New?' button to the dropdown (#14379) chore(recordings): don't DLQ on PostgreSQL errors (#14438) chore: update autocapture attribute capture (#14435) chore(recordings): remove hub dependency on recordings ingestion (#14418) chore(deps): Update posthog-js to 1.49.0 (#14436) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hub is a grab bag of depencencies that are not all required for
recordings ingestion. To keep the recordings ingestion lean, we
remove the hub dependency and use the postgres and kafka client
directly.
This should increase the availability of the session recordings
workload, e.g. it should not go down it Redis or ClickHouse is down.
The driver behind this is that we're doing a whole load of things on the
recording ingestion pods that we don't need to. It hurts startup times
and as a result autoscaling. And it hurts e.g. resources on ClickHouse
and Redis. It hurts availability of the recordings ingestion.
I've also added a healthcheck for session recordings that will mean that
recordings pods are restarted if they get into trouble ingesting.
Note that I have refactored some of the db.ts to call out to specific calls
in e.g. teamManager, which should have the most knowledge of what teams
look like in the database, rather than including it in the catch all db. there
isn't much benefit to decoupling that from the teamManager and just means
that it is confusing to follow the code paths related to it.
Problem
Changes
👉 Stay up-to-date with PostHog coding conventions for a smoother review.
How did you test this code?