feat: otlp allow not translating UTF8 characters by jmichalek132 · Pull Request #14974 · prometheus/prometheus

jmichalek132 · 2024-09-24T23:41:54Z

ywwg · 2024-09-25T19:03:22Z

config/config.go

 // OTLPConfig is the configuration for writing to the OTLP endpoint.
 type OTLPConfig struct {
 	PromoteResourceAttributes []string `yaml:"promote_resource_attributes,omitempty"`
+	TranslateDots             bool     `default:"true" yaml:"translate_dots"`


as said in the meeting, let's make this an Enum so that there is future-proofing for other possible translation modes. Something like "NameConversionMode" with values of "NoConversion" and "ClassicPrometheusConverstion"

ArthurSens · 2024-09-26T18:11:11Z

@jmichalek132 as discussed in a few calls this week, let me take over this PR so you can focus on your mentorship project :)

@aknuds1 @ywwg, I'm wondering what we want to accomplish here. There are two things on my mind:

OTLP users want their metrics as is. No underscores, no units, no nothing, and we want to allow this use case.
Before 3.0 Prometheus did not accept UTF8 characters by default so we had to translate UTF8 characters to underscores. Now it's a different scenario, UTF8 is accepted by default and we can accept them as they are. (No need to worry about unit suffixes)

Which of those problems are we solving here? Depending on the answer I'm thinking about two different API for the config file:

otlp:
  otlp_to_prometheus_translation: NoTranslation (default) | ClassicPrometheusTranslation (UTF8 to underscore + unit suffixes)

otlp:
  add_unit_suffixes: false (default) | true #Probably not being worked on in this PR
  utf8_to_underscore: false (default) | true

jmichalek132 · 2024-09-27T09:05:10Z

@jmichalek132 as discussed in a few calls this week, let me take over this PR so you can focus on your mentorship project :)

@aknuds1 @ywwg, I'm wondering what we want to accomplish here. There are two things on my mind:

OTLP users want their metrics as is. No underscores, no units, no nothing, and we want to allow this use case.

Before 3.0 Prometheus did not accept UTF8 characters by default so we had to translate UTF8 characters to underscores. Now it's a different scenario, UTF8 is accepted by default and we can accept them as they are. (No need to worry about unit suffixes)

Which of those problems are we solving here? Depending on the answer I'm thinking about two different API for the config file:
otlp:
  otlp_to_prometheus_translation: NoTranslation (default) | ClassicPrometheusTranslation (UTF8 to underscore + unit suffixes)
otlp:
  add_unit_suffixes: false (default) | true #Probably not being worked on in this PR
  utf8_to_underscore: false (default) | true

For reference linking the consensus from the Prometheus in person dev summit on this topic

https://docs.google.com/document/d/1uurQCi5iVufhYHGlBZ8mJMK_freDFKPG0iYBQqJ9fvA/edit#bookmark=id.x5wujq3bsjjo

CONSENSUS: We recommend keeping the metric names from OTel SDKs as unchanged as possible when converting to Prometheus samples as the default.
DISAGREE, but not blocking Fabian Stäber and Julius Volz would like to keep units and _total
CONSENSUS: For classic histograms we will still add _bucket, _count and _sum,similar for summaries.
CONSENSUS: We intend for OpenMetrics 2.x to support this, targeting 2.0
CONSENSUS: We will have a Prometheus mode which converts . to _ with suffixes and units.

aknuds1 · 2024-09-27T09:23:37Z

@aknuds1 @ywwg, I'm wondering what we want to accomplish here. There are two things on my mind:

@ArthurSens @jmichalek132 @ywwg given the consensus in the latest dev summit ("We want to explore ways to do type and unit annotations in PromQL"), we've concluded that we need a way to ensure time series are identified through type and unit, right? So we avoid collisions between metrics that are named the same, but have different types/units.

aknuds1 · 2024-09-27T09:26:56Z

I think the PR needs to define properly which problem is being solved. Jumping into a solution isn't the best approach.

ArthurSens · 2024-09-27T10:17:59Z

I think the PR needs to define properly which problem is being solved. Jumping into a solution isn't the best approach.

Yes exactly! I was hoping we could come to an agreement before I continue with the PR :)

aknuds1 · 2024-09-27T10:33:16Z

Yes exactly! I was hoping we could come to an agreement before I continue with the PR :)

Agree, we need to decide on a solution to the problem laid out in OpenTelemetry → Prometheus: type+unit suffixes, right?

The agreed upon solution would have to be well defined, which I think we haven't got to yet.

ArthurSens · 2024-09-27T12:14:06Z

Agree, we need to decide on a solution to the problem laid out in OpenTelemetry → Prometheus: type+unit suffixes, right?

The agreed upon solution would have to be well defined, which I think we haven't got to yet.

If that's a requirement for the work here, I'm afraid this PR will have to wait until Prometheus 4.0 :/

I don't think we'll manage to implement any of those proposals before 3.0 (scheduled for november this year as far as I could understand). So I was hoping we could find a middle ground: let OTel users have what they want, even if not the best solution.

If we release 3.0 with the current behavior (doing translations), changing the default to no translation would be a breaking change. Could we release 3.0 with no translation, as far as it's well documented in the OTLP guide that name colisions are possible and people should opt-in to ClassicPrometheusTranslation to avoid these situations? Once we have good solution for units/types, we can finally drop the warning in our guides

ArthurSens · 2024-09-27T12:16:54Z

If the team decides that we really want to wait for proper unit/type handling, I guess I could continue the PR... just changing the default to ClassicPrometheusTranslation

aknuds1 · 2024-09-27T14:38:45Z

If that's a requirement for the work here, I'm afraid this PR will have to wait until Prometheus 4.0 :/

Could you start a discussion with @gouthamve in CNCF Slack on what his current stance is wrt. dropping OTel metric name translation? I kind of forgot what he last communicated.

ywwg · 2024-09-27T14:42:35Z

My sense from the Otel folks is that the conversion to underscores is a pain point right now, so I'd prefer to find a way to allow the default to be no-translation. If my understanding is correct the issue of same-name-different-type has only been hypothetical so far. I think the benefit to the otel community of having names pass through unchanged outweighs the unproven possibility of collisions.

Perhaps it would be sufficient to add a note in the documentation explaining "by default no translation is done to Otel metric names. If there are multiple metrics with the same name but different types or units, these metrics will collide when queried in Prometheus. In this situation, administrators should set the translation mode to ClassicPrometheusTranslation to avoid the collisions".

As for config choices:

otlp:
  add_unit_suffixes: false (default) | true #Probably not being worked on in this PR
  utf8_to_underscore: false (default) | true

if we do this, does add_unit_suffixes always append with an underscore? imho I think it might be fine to use a dot if the user has otherwise not specified underscore conversion, e.g.: "http.status" → "http.status.total". Because of ambiguities like this, I think I'm more in favor of the enum approach:

otlp:
  otlp_to_prometheus_translation: NoTranslation (default) | ClassicPrometheusTranslation (UTF8 to underscore + unit suffixes)

aknuds1 · 2024-09-29T11:05:04Z

@ywwg @ArthurSens in the latest dev summit, @dashpole noted the following about @gouthamve's latest thinking:

After the inperson meeting Goutham realized that it’s not as easy as initially thought. We need to account for metric name collision if we don’t handle unit and types somehow.

Doesn't this indicate we need a solution that avoids collisions (as our current solution does, through suffixes)?

ArthurSens · 2024-09-30T14:02:46Z

After the inperson meeting Goutham realized that it’s not as easy as initially thought. We need to account for metric name collision if we don’t handle unit and types somehow.

Doesn't this indicate we need a solution that avoids collisions (as our current solution does, through suffixes)?

For the perfect scenario, yes! I was just hoping that we could release something that allows no translation before 3.0, even knowing that name collisions is a possibility, while we keep working on a long term solution that will take a few months to release.

Would you be ok if we continue this PR but change the default behavior to keep translating things? NoTranslation would be opt-in, while being very well documented that it can lead to name collision.

aknuds1 · 2024-09-30T14:04:55Z

For the perfect scenario, yes! I was just hoping that we could release something that allows no translation before 3.0, even knowing that name collisions is a possibility, while we keep working on a long term solution that will take a few months to release.

I'm kind of agnostic, I'm just trying to figure out what is the actual consensus in this sense. I've asked @gouthamve in CNCF Slack. I primarily want to avoid adding changes we end up not wanting. @dashpole's note from the last summit gives the impression we don't want a translation mode that can lead to collisions.

jmichalek132 · 2024-09-30T16:23:09Z

My sense from the Otel folks is that the conversion to underscores is a pain point right now, so I'd prefer to find a way to allow the default to be no-translation. If my understanding is correct the issue of same-name-different-type has only been hypothetical so far. I think the benefit to the otel community of having names pass through unchanged outweighs the unproven possibility of collisions.

Perhaps it would be sufficient to add a note in the documentation explaining "by default no translation is done to Otel metric names. If there are multiple metrics with the same name but different types or units, these metrics will collide when queried in Prometheus. In this situation, administrators should set the translation mode to ClassicPrometheusTranslation to avoid the collisions".

As for config choices:
otlp:
  add_unit_suffixes: false (default) | true #Probably not being worked on in this PR
  utf8_to_underscore: false (default) | true
if we do this, does add_unit_suffixes always append with an underscore? imho I think it might be fine to use a dot if the user has otherwise not specified underscore conversion, e.g.: "http.status" → "http.status.total". Because of ambiguities like this, I think I'm more in favor of the enum approach:
otlp:
  otlp_to_prometheus_translation: NoTranslation (default) | ClassicPrometheusTranslation (UTF8 to underscore + unit suffixes)

I agree with @ywwg suggestions.
Also to add my 2 cents,

It feels like a pity to have all the work done to support UTF-8 in Prometheus and then preventing people from utilizing it by not allowing to turn off the translation.
The collisions when not adding suffixes can already happen when using remote write exporter in otel collector given it has an option to disable adding the suffixes https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/README.md but imho this should be very rare.
Whatever we do here I would prefer if we stick to the same approach in the prometheus exporters in otel collector / otel sdks.

ArthurSens · 2024-09-30T18:57:18Z

I've updated the PR to not translate UTF8 characters by default. I'm also not touching unit/type suffixes in any way!

I'm happy to change the PR again to allow not adding unit/type suffixes, but I don't want to block the UTF8 translation just because we can't find an agreement on unit/type handling

ywwg · 2024-10-22T19:49:39Z

config/config.go

+type translationStrategyOption string
+
+var (
+	// NoUTF8EscapingWithSuffixes will keep UTF-8 characters as they are, units and type suffixes will still be added.


I thought the two options were going to be "no alteration at all" and "underscoreescapingwithsuffixes"?

(is this because type/unit is differentiating?)

ywwg · 2024-10-22T19:53:34Z

I am fine with adding an experimental mode to not translate provided we make it clear that this is experimental and can break in the future. Should we do it via a feature-flag or not needs to be decided.

context for why I thought we were also going to have a no-translation mode with caveats

ywwg · 2024-10-23T15:24:27Z

Per our discussion, because units are differentiating and prometheus does not support otel unit metadata well, it makes sense for prom's otel endpoint to add suffixes in all available modes. In the future when prom does support unit metadata, we can revisit the enum and add a mode to pass names unaltered.

aknuds1 · 2024-10-30T16:51:24Z

storage/remote/otlptranslator/prometheus/normalize_label.go

 	}

-	// Replace all non-alphanumeric runes with underscores
-	label = strings.Map(sanitizeRune, label)


Why do you no longer use this method (for the non-allowUTF8 case), and why do call model.EscapeName(label, model.UnderscoreEscaping) (assuming this is supposed to have the same effect?) after potentially prefixing with key?

I basically don't understand why we would change the old mode (i.e. where we convert UTF8 chars to underscores).

I'm using model.EscapeName as a suggestion from @ywwg!

The previous function isn't working as expected, it translates characters to underscores if unicode.IsLetter(r) || unicode.IsDigit(r) is not true.

This is problematic because:

Foreign letters like russian, japanese, german complies with unicode.IsLetter() and they are being replaced when they shouldn't

The character : is valid even without UTF8 and is being replaced. See otlp writer: colons (':') in metric names replaced with underscores ('_') #15199

It looks impractical to concatenate every foreign character in the sanitizeRunes conditional, so I'm just using the same function used in prometheus/common like owen suggested :)

I'm using model.EscapeName as a suggestion from @ywwg!

If we are making changes unrelated to the PR's topic (add un-escaped UTF8 mode), it should be a separate PR. I want to see explicitly what the effect is from using model.EscapeName instead.

This PR was now rebased on top of #15249, so we need that one merged first

aknuds1

I don't think tests should loop over true/false for allowUTF8, there should instead be one sub-test for each mode.

Also see my previous question regarding NormalizeLabel.

aknuds1 · 2024-10-30T17:00:17Z

storage/remote/otlptranslator/prometheus/normalize_name_test.go

+	for _, allowUTF8 := range []bool{true, false} {
+		if allowUTF8 {
+			require.Equal(t, "unsupported.metric.temperature_°F", normalizeName(createGauge("unsupported.metric.temperature", "°F"), "", allowUTF8))
+			require.Equal(t, "unsupported.metric.weird_+=.:,!* & #", normalizeName(createGauge("unsupported.metric.weird", "+=.:,!* & #"), "", allowUTF8))
+			require.Equal(t, "unsupported.metric.redundant___test $_per_°C", normalizeName(createGauge("unsupported.metric.redundant", "__test $/°C"), "", allowUTF8))
+		} else {
+			require.Equal(t, "unsupported_metric_temperature_F", normalizeName(createGauge("unsupported.metric.temperature", "°F"), "", allowUTF8))
+			require.Equal(t, "unsupported_metric_weird", normalizeName(createGauge("unsupported.metric.weird", "+=.:,!* & #"), "", allowUTF8))
+			require.Equal(t, "unsupported_metric_redundant_test_per_C", normalizeName(createGauge("unsupported.metric.redundant", "__test $/°C"), "", allowUTF8))
+		}
+	}


This shouldn't be a loop.

Suggested change

for _, allowUTF8 := range []bool{true, false} {

if allowUTF8 {

require.Equal(t, "unsupported.metric.temperature_°F", normalizeName(createGauge("unsupported.metric.temperature", "°F"), "", allowUTF8))

require.Equal(t, "unsupported.metric.weird_+=.:,!* & #", normalizeName(createGauge("unsupported.metric.weird", "+=.:,!* & #"), "", allowUTF8))

require.Equal(t, "unsupported.metric.redundant___test $_per_°C", normalizeName(createGauge("unsupported.metric.redundant", "__test $/°C"), "", allowUTF8))

} else {

require.Equal(t, "unsupported_metric_temperature_F", normalizeName(createGauge("unsupported.metric.temperature", "°F"), "", allowUTF8))

require.Equal(t, "unsupported_metric_weird", normalizeName(createGauge("unsupported.metric.weird", "+=.:,!* & #"), "", allowUTF8))

require.Equal(t, "unsupported_metric_redundant_test_per_C", normalizeName(createGauge("unsupported.metric.redundant", "__test $/°C"), "", allowUTF8))

}

}

t.Run("allow UTF8", func(t *testing.T) {

require.Equal(t, "unsupported.metric.temperature_°F", normalizeName(createGauge("unsupported.metric.temperature", "°F"), "", allowUTF8))

require.Equal(t, "unsupported.metric.weird_+=.:,!* & #", normalizeName(createGauge("unsupported.metric.weird", "+=.:,!* & #"), "", allowUTF8))

require.Equal(t, "unsupported.metric.redundant___test $_per_°C", normalizeName(createGauge("unsupported.metric.redundant", "__test $/°C"), "", allowUTF8))

})

t.Run("disallow UTF8", func(t *testing.T) {

require.Equal(t, "unsupported_metric_temperature_F", normalizeName(createGauge("unsupported.metric.temperature", "°F"), "", allowUTF8))

require.Equal(t, "unsupported_metric_weird", normalizeName(createGauge("unsupported.metric.weird", "+=.:,!* & #"), "", allowUTF8))

require.Equal(t, "unsupported_metric_redundant_test_per_C", normalizeName(createGauge("unsupported.metric.redundant", "__test $/°C"), "", allowUTF8))

})

storage/remote/otlptranslator/prometheus/normalize_name_test.go

storage/remote/write_handler.go

aknuds1 · 2024-10-30T17:15:06Z

storage/remote/write_handler.go

 	annots, err := converter.FromMetrics(r.Context(), req.Metrics(), otlptranslator.Settings{
 		AddMetricSuffixes:         true,
 		PromoteResourceAttributes: otlpCfg.PromoteResourceAttributes,
+		AllowUTF8:                 (model.NameValidationScheme == model.UTF8Validation) && (otlpCfg.TranslationStrategy == config.NoUTF8EscapingWithSuffixes),


It concerns me that if model.NameValidationScheme is not model.UTF8Validation, the NoUTF8EscapingWithSuffixes translation strategy is silently ignored. How about we instead log a warning if model.NameValidationScheme != model.UTF8Validation && otlpCfg.TranslationStrategy == config.NoUTF8EscapingWithSuffixes, while also setting AllowUTF8: false?

Good point!

I think we can warn that while loading the configuration

aknuds1

In retrospect, I don't think CleanUpString should change. Just don't use it when allowing UTF8.

storage/remote/otlptranslator/prometheus/normalize_name.go

Signed-off-by: Arthur Silva Sens <[email protected]>

Signed-off-by: Juraj Michalek <[email protected]>

Signed-off-by: Arthur Silva Sens <[email protected]>

ArthurSens · 2024-10-30T22:42:43Z

Folks, I've messed up my commits here 😓. Let's get this one merged first and I'll figure out what to do here later

ArthurSens · 2024-10-31T22:07:11Z

Lol ok, after merging the chained PR the branch got automatically deleted and messed up this PR. I'll open another one, sorry everyone

jmichalek132 force-pushed the jm-feat-otlp-allow-not-translating-dots branch from 2cda163 to 67b9038 Compare September 24, 2024 23:43

ywwg mentioned this pull request Sep 25, 2024

Epic: Implement UTF-8 character support for label and metric names #13095

Closed

40 tasks

ywwg reviewed Sep 25, 2024

View reviewed changes

ArthurSens mentioned this pull request Sep 26, 2024

otlp endpoint: No metric translation by default #14990

Open

ArthurSens force-pushed the jm-feat-otlp-allow-not-translating-dots branch 2 times, most recently from 3af6071 to d0a868c Compare September 30, 2024 18:46

ArthurSens changed the title ~~feat: otlp allow not translating dots~~ feat: otlp allow not translating UTF8 characters Sep 30, 2024

ArthurSens force-pushed the jm-feat-otlp-allow-not-translating-dots branch from d0a868c to 4e03883 Compare September 30, 2024 18:54

ArthurSens force-pushed the jm-feat-otlp-allow-not-translating-dots branch from 4e03883 to 4796f57 Compare September 30, 2024 19:02

ArthurSens marked this pull request as ready for review September 30, 2024 19:18

ArthurSens requested review from aknuds1, bwplotka, cstyan, jesusvazquez and tomwilkie as code owners September 30, 2024 19:18

dashpole previously approved these changes Oct 21, 2024

View reviewed changes

ywwg reviewed Oct 22, 2024

View reviewed changes

ywwg previously approved these changes Oct 23, 2024

View reviewed changes

ArthurSens mentioned this pull request Oct 28, 2024

otlp writer: colons (':') in metric names replaced with underscores ('_') #15199

Closed

aknuds1 reviewed Oct 30, 2024

View reviewed changes

aknuds1 requested changes Oct 30, 2024

View reviewed changes

aknuds1 reviewed Oct 30, 2024

View reviewed changes

storage/remote/otlptranslator/prometheus/normalize_name.go Show resolved Hide resolved

storage/remote/otlptranslator/prometheus/normalize_name.go Show resolved Hide resolved

storage/remote/otlptranslator/prometheus/normalize_name.go Show resolved Hide resolved

otlp translator: Add test showing bugs

39f4035

Signed-off-by: Arthur Silva Sens <[email protected]>

ArthurSens mentioned this pull request Oct 30, 2024

bugfix: Fix otlp translator for foreign characters #15249

Merged

ArthurSens and others added 6 commits October 30, 2024 18:16

bugfix: Fix otlp translator for colons and foreign characters

a4b1487

Signed-off-by: Arthur Silva Sens <[email protected]>

feat: otlp allow not translating dots

91c3149

Signed-off-by: Juraj Michalek <[email protected]>

chore: updated unit tests

1484007

Signed-off-by: Juraj Michalek <[email protected]>

chore: fix unused metadata function

3d0e13f

Signed-off-by: Juraj Michalek <[email protected]>

chore: updated unit tests, missed one before

bd5a8af

Signed-off-by: Juraj Michalek <[email protected]>

Allow UTF-8 characters in metric and label names as opt-in feature

e1158ca

Signed-off-by: Arthur Silva Sens <[email protected]>

ArthurSens dismissed stale reviews from ywwg and dashpole via e1158ca October 30, 2024 21:26

ArthurSens force-pushed the jm-feat-otlp-allow-not-translating-dots branch from edf84a8 to e1158ca Compare October 30, 2024 21:26

ArthurSens changed the base branch from main to otlp-fix-sanitiza October 30, 2024 21:28

ArthurSens force-pushed the otlp-fix-sanitiza branch 3 times, most recently from 8df09f3 to ea06f1a Compare October 30, 2024 22:29

ArthurSens deleted the branch prometheus:otlp-fix-sanitiza October 31, 2024 22:05

ArthurSens closed this Oct 31, 2024

ArthurSens mentioned this pull request Nov 1, 2024

Allow UTF-8 characters in metric and label names as opt-in feature #15258

Merged

jmichalek132 deleted the jm-feat-otlp-allow-not-translating-dots branch February 15, 2025 11:44

Conversation

jmichalek132 commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ywwg Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

ArthurSens commented Sep 26, 2024

Uh oh!

jmichalek132 commented Sep 27, 2024

Uh oh!

aknuds1 commented Sep 27, 2024

Uh oh!

aknuds1 commented Sep 27, 2024

Uh oh!

ArthurSens commented Sep 27, 2024

Uh oh!

aknuds1 commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArthurSens commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArthurSens commented Sep 27, 2024

Uh oh!

aknuds1 commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ywwg commented Sep 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aknuds1 commented Sep 29, 2024

Uh oh!

ArthurSens commented Sep 30, 2024

Uh oh!

aknuds1 commented Sep 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmichalek132 commented Sep 30, 2024

Uh oh!

ArthurSens commented Sep 30, 2024

Uh oh!

ywwg Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

ywwg Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

ywwg commented Oct 22, 2024

Uh oh!

ywwg commented Oct 23, 2024

Uh oh!

aknuds1 Oct 30, 2024

Choose a reason for hiding this comment

Uh oh!

ArthurSens Oct 30, 2024

Choose a reason for hiding this comment

Uh oh!

aknuds1 Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurSens Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aknuds1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aknuds1 Oct 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aknuds1 Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurSens Oct 30, 2024

jmichalek132 commented Sep 24, 2024 •

edited

Loading

aknuds1 commented Sep 27, 2024 •

edited

Loading

ArthurSens commented Sep 27, 2024 •

edited

Loading

aknuds1 commented Sep 27, 2024 •

edited

Loading

ywwg commented Sep 27, 2024 •

edited

Loading

aknuds1 commented Sep 30, 2024 •

edited

Loading

aknuds1 Oct 30, 2024 •

edited

Loading

ArthurSens Oct 30, 2024 •

edited

Loading

aknuds1 left a comment •

edited

Loading

aknuds1 Oct 30, 2024 •

edited

Loading