* Use the same healthz

Signed-off-by: Elena Kolevska <[email protected]>

* new healthz

Signed-off-by: Elena Kolevska <[email protected]>

* Increase test timeout

Signed-off-by: Elena Kolevska <[email protected]>

* kit: bump to latest main

Signed-off-by: Luis Rascao <[email protected]>

* Actors: State store actor reminder drain grace

Update state store actor reminder engine to lock evaluation during
drain events, and give a 2 second grace period concurrently to all
currently active reminders to complete before completing dissemination.

Signed-off-by: joshvanl <[email protected]>

* Give not grace period to reminder stopCh

Signed-off-by: joshvanl <[email protected]>

* Workflow: default max parallel items to 1000 (#8432)

* Workflow: default max parallel items to 1000

Up the number of work items which can be run in parallel from 100 to
1000

Signed-off-by: joshvanl <[email protected]>

* Fix unit test for new default max parallel work items value

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* Always update state reminder stack, and add wg wait

Signed-off-by: joshvanl <[email protected]>

* Ensure tick is recorded on every execution

Signed-off-by: joshvanl <[email protected]>

* Update durabletask-go to head

Signed-off-by: joshvanl <[email protected]>

* Workflow: Remove actor table memory leak (#8429)

Remove memory leak whereby workflow actor objects were not be pruned
from the actor table slice.

After an activity fires, delete the object from the actor table.

When a workflow reaches a terminal state or is purged, remove it from
the actor table.

Signed-off-by: joshvanl <[email protected]>

* Fixes deadlock

Signed-off-by: Elena Kolevska <[email protected]>

* Disable streamhang test on windows

Signed-off-by: Elena Kolevska <[email protected]>

* Scheduler: ETCD mTLS

Updates scheduler for ETCD to run with mTLS between hosts. Updates
sentry to make dapr-scheduler identity certificates to be signed with
`dapr-scheduler.${ns}.svc` DNS. Updates ETCD embeded config to use mTLS
on this DNS when mTLS is enabled.

Signed-off-by: joshvanl <[email protected]>

* Update dapr/kit to HEAD

Signed-off-by: joshvanl <[email protected]>

* Fix etcd endpoint for chart

Signed-off-by: joshvanl <[email protected]>

* test fixup

Signed-off-by: joshvanl <[email protected]>

* Use correct scheduler service name in Scheduler identity DNS SAN

Signed-off-by: joshvanl <[email protected]>

* Match on correct Scheduler host name

Signed-off-by: joshvanl <[email protected]>

* Use correct headless service, and delete HTTP client port

Signed-off-by: joshvanl <[email protected]>

* Fix helm int test

Signed-off-by: joshvanl <[email protected]>

* Update scheduler app options

Signed-off-by: joshvanl <[email protected]>

* Actors: Skip placement lock on local reentrance (#8438)

* Actors: Skip placement lock on local reentrance

When in a local actor reentrancy call, skip placement lock so that a
local actor chain call succeeds to completion before releasing the lock
for dissemination.

Signed-off-by: joshvanl <[email protected]>

* Set internal `Dapr-API-Call` to ensure we don't skip placement lock if
reentrancy ID is supplied by first client call

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* Update diagridio/go-etcd-cron to head (#8444)

Signed-off-by: joshvanl <[email protected]>

* Docs and code fixes

Signed-off-by: Elena Kolevska <[email protected]>

* patches

Signed-off-by: Elena Kolevska <[email protected]>

* Workflow: dont delete reminder on placement dissemination (#8445)

Do not mark the workflow orchestration or activity reminder for
deletion when placement dissemination events occurs.

Signed-off-by: joshvanl <[email protected]>

* patch : integration-sidecar-master

Signed-off-by: Elena Kolevska <[email protected]>

* Scheduler: reuse client stream (#8439)

* Scheduler: reuse client stream

Reuse the same scheduler stream in the watch hosts client loop.

Signed-off-by: joshvanl <[email protected]>

* Lint

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* More patches

Signed-off-by: Elena Kolevska <[email protected]>

* Remove empty file

Signed-off-by: joshvanl <[email protected]>

* Removes workflow backend

Signed-off-by: Elena Kolevska <[email protected]>

* Removes sql patch

Signed-off-by: Elena Kolevska <[email protected]>

* patch update

Signed-off-by: Elena Kolevska <[email protected]>

* More patches

Signed-off-by: Elena Kolevska <[email protected]>

* Workflow: Terminate sync API not found (#8443)

* Workflow: Terminate sync API not found

Also allow the Workflow to be considered Terminated once the Workflow
metadata can no longer be found.

Signed-off-by: joshvanl <[email protected]>

* Lint

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Cassie Coyle <[email protected]>

* Workflow: Cache workflow state for 10s after Termination (#8447)

Delay deleting the workflow orchestration actor from being deleted from
the actor table for 10s in order to hold the cache for new streaming
clients to observe the state.

10s is enough, as the client is Terminating the workflow anyway.

Signed-off-by: joshvanl <[email protected]>

* Actors: Move reentrancy lock to top level (#8449)

* Actors: Move reentrancy lock to top level

Moves the reentrnacy actor lock to the top level of the API call stack
in the engine. This is done so that in the event of a placement
dissemination during 2 calls of the same actor ID, after unlocking the
order of calls is preserved and there is not a non-deterministic race to
when either call is processed.

Introduces a `Locker` which is responsible for indexing actor ID locks.
Workflow activity actor reminders will continue to skip locks for
reminders when Scheduler push based reminders are used.

Fixes actor lock e2e test.

Signed-off-by: joshvanl <[email protected]>

* Backoff permanently from local actor call

Signed-off-by: joshvanl <[email protected]>

* Lint

Signed-off-by: joshvanl <[email protected]>

* Fix actor top level resiliency by no propagating backoff when policy
defined

Signed-off-by: joshvanl <[email protected]>

* Adds gate to workflow backend orchestration create until types are fully
registered

Signed-off-by: joshvanl <[email protected]>

* Cleanup return errors

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* Final int  test fixes

Signed-off-by: Elena Kolevska <[email protected]>

* corrected patch

Signed-off-by: Elena Kolevska <[email protected]>

* Missed line in patch

Signed-off-by: Elena Kolevska <[email protected]>

* Removes logApplyConcurrency

Signed-off-by: Elena Kolevska <[email protected]>

* Fixes test

Signed-off-by: Elena Kolevska <[email protected]>

* Linter

Signed-off-by: Elena Kolevska <[email protected]>

* Improve test stability (#8399)

* handle partial state loads in workflow actor

Signed-off-by: Cassandra Coyle <[email protected]>

* add comment

Signed-off-by: Cassandra Coyle <[email protected]>

* log appname

Signed-off-by: Cassandra Coyle <[email protected]>

* up period on reminder due to slow CI

Signed-off-by: Cassandra Coyle <[email protected]>

* cleanup reminders

Signed-off-by: Cassandra Coyle <[email protected]>

* clean up reminders

Signed-off-by: Cassandra Coyle <[email protected]>

* add check on repeats left for statestore reminders

Signed-off-by: Cassandra Coyle <[email protected]>

* Update tests/e2e/actor_features/actor_features_test.go

Co-authored-by: Josh van Leeuwen <[email protected]>
Signed-off-by: Cassie Coyle <[email protected]>

* reset wf state changes

Signed-off-by: Cassandra Coyle <[email protected]>

* try to rm the ability for additional triggers

Signed-off-by: Cassandra Coyle <[email protected]>

* rm check for expiration

Signed-off-by: Cassandra Coyle <[email protected]>

* tweak

Signed-off-by: Cassandra Coyle <[email protected]>

* reset reminder state changes

Signed-off-by: Cassandra Coyle <[email protected]>

* debug

Signed-off-by: Cassandra Coyle <[email protected]>

* check repeats left before decrementing

Signed-off-by: Cassandra Coyle <[email protected]>

* is this it?

Signed-off-by: Cassandra Coyle <[email protected]>

* rm debug logs

Signed-off-by: Cassandra Coyle <[email protected]>

* rm debugging changes

Signed-off-by: Cassandra Coyle <[email protected]>

* rm sleep

Signed-off-by: Cassandra Coyle <[email protected]>

* fix unit tests

Signed-off-by: Cassandra Coyle <[email protected]>

* fix unit tests bc this check is not needed due to the check before calling the reminder if repeats == 0

Signed-off-by: Cassandra Coyle <[email protected]>

* fingers crossed this fixes all cases

Signed-off-by: Cassandra Coyle <[email protected]>

* special case wf timers

Signed-off-by: Cassandra Coyle <[email protected]>

* fix a few int tests that failed locally & try fixing e2e

Signed-off-by: Cassandra Coyle <[email protected]>

* rebase

Signed-off-by: Cassandra Coyle <[email protected]>

* try this in CI

Signed-off-by: Cassandra Coyle <[email protected]>

* Race condition write to err var in tests
Renamed shadowed var reminderName

Signed-off-by: Anton Troshin <[email protected]>

* Actor features tests were restarting the app and loosing connection to portforward when running locally.
Re-establish port forward and refresh URLs
Error format fix

Signed-off-by: Anton Troshin <[email protected]>

* Uncomment logURL update

Signed-off-by: Anton Troshin <[email protected]>

* Add another url refresh after restart in test

Signed-off-by: Anton Troshin <[email protected]>

* try using range which is thread safe

Signed-off-by: Cassandra Coyle <[email protected]>

* fix build issues since my upstream was a commit behind locally

Signed-off-by: Cassandra Coyle <[email protected]>

* try using evaluationChan while draining reminders

Signed-off-by: Cassandra Coyle <[email protected]>

* debug

Signed-off-by: Cassandra Coyle <[email protected]>

* lock on the gettrack call and add back in other tests since that seems to bring up the issue vs when they are removed and there are less reminders

Signed-off-by: Cassandra Coyle <[email protected]>

* rm debug logs from statestore reminder

Signed-off-by: Cassandra Coyle <[email protected]>

* reset debug lines + comment out other e2e tests that consistently pass for a quicker round trip

Signed-off-by: Cassandra Coyle <[email protected]>

* reset test mk file

Signed-off-by: Cassandra Coyle <[email protected]>

* fix int flake on metrics

Signed-off-by: Cassandra Coyle <[email protected]>

* more locking on reminder track

Signed-off-by: Cassandra Coyle <[email protected]>

* rlock -> lock

Signed-off-by: Cassandra Coyle <[email protected]>

* debug logs, rm reminder lock

Signed-off-by: Cassandra Coyle <[email protected]>

* switch to cmap and rm map & lock

Signed-off-by: Cassandra Coyle <[email protected]>

* fix more metrics flakes

Signed-off-by: Cassandra Coyle <[email protected]>

* make lint

Signed-off-by: Cassandra Coyle <[email protected]>

* rm debug logs

Signed-off-by: Cassandra Coyle <[email protected]>

* revert back to old map for pref for speed since race is fixed

Signed-off-by: Cassandra Coyle <[email protected]>

* rm diff

Signed-off-by: Cassandra Coyle <[email protected]>

* cleanup of diff

Signed-off-by: Cassandra Coyle <[email protected]>

* cleanup

Signed-off-by: Cassandra Coyle <[email protected]>

* cleanup

Signed-off-by: Cassandra Coyle <[email protected]>

* cleanup

Signed-off-by: Cassandra Coyle <[email protected]>

* oops didn't have latest

Signed-off-by: Cassandra Coyle <[email protected]>

* fix flaky int test metrics call

Signed-off-by: Cassandra Coyle <[email protected]>

---------

Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Cassie Coyle <[email protected]>
Signed-off-by: Anton Troshin <[email protected]>
Co-authored-by: Josh van Leeuwen <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>
Co-authored-by: Anton Troshin <[email protected]>

* Async error checking

Signed-off-by: Elena Kolevska <[email protected]>

* Workflow: Always return errors with Scheduler enabled

Always return an error from a Scheduler reminder enabled workflow
execution (context cancels) to ensure that the reminder is not marked
for deletion and the event is retired.

Signed-off-by: joshvanl <[email protected]>

* Workflow: Return input & output from Get (#8451)

* Workflow: Return input & output from Get

Update Workflow universal Get response to include the input & output
values in the Properties, as they were before.

Adds int tests.

Signed-off-by: joshvanl <[email protected]>

* Adds resiliency to workflow start workflow call

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* go.mod: Update go-etcd-cron to HEAD (#8453)

* go.mod: Update go-etcd-cron to HEAD

Updates go-etcd-cron to head. Include an int test to ensure a client can
create a job with a schedule with 6 columns in cron style.

Signed-off-by: joshvanl <[email protected]>

* go mod tidy

Signed-off-by: joshvanl <[email protected]>

* Updates go-etcd-cron to v0.4.0

Signed-off-by: joshvanl <[email protected]>

* Update durabletask-go v0.6.0 for vanity

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* Actors: Regression: return 200 on reminder not found (#8456)

* Actors: Regression: return 200 on reminder not found

Update Get Actor Reminder to return a 200 when Scheduler reminders are
enabled. This matches the same behavior as the state store reminder get
request.

Adds integration test.

Signed-off-by: joshvanl <[email protected]>

* Update error to debug log

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* Extend test timeout

Signed-off-by: Elena Kolevska <[email protected]>

* More error code tests fixes

Signed-off-by: Elena Kolevska <[email protected]>

One more test fix

Signed-off-by: Elena Kolevska <[email protected]>

* Async error checking

Signed-off-by: Elena Kolevska <[email protected]>

* More error code tests fixes

Signed-off-by: Elena Kolevska <[email protected]>

One more test fix

Signed-off-by: Elena Kolevska <[email protected]>

* Add raft leadership in test

Signed-off-by: Elena Kolevska <[email protected]>

* Revert "Async error checking"

This reverts commit 07e4eadf5d756417349392deb55deb52b8e16f71.

Signed-off-by: Elena Kolevska <[email protected]>

* Revert "More error code tests fixes"

This reverts commit 7864a768332a72a2eba13fcd8abbc8663159d475.

Signed-off-by: Elena Kolevska <[email protected]>

* Workflow: remove reminder policy on remote actor call

Signed-off-by: joshvanl <[email protected]>

* Integration: More test stabilisations (#8455)

* Integration: More test stabilisations

Signed-off-by: joshvanl <[email protected]>

* Use assert over require

Signed-off-by: joshvanl <[email protected]>

* Increase start-up check timeout

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* diagridio/go-etcd-cron: v0.4.1

Signed-off-by: joshvanl <[email protected]>

* Updates CRDs

Signed-off-by: joshvanl <[email protected]>

* Bring back subscription conversion webhook block

Signed-off-by: joshvanl <[email protected]>

* Increases timeouts

Signed-off-by: Elena Kolevska <[email protected]>

* Adds actors scheduler reminders patch

Signed-off-by: Elena Kolevska <[email protected]>

* Fix Scheduler readOnlyRootFilesystem: true (#8465)

* test readOnlyRootFilesystem: true

Signed-off-by: Cassandra Coyle <[email protected]>

* test readOnlyRootFilesystem: false

Signed-off-by: Cassandra Coyle <[email protected]>

* hard code true

Signed-off-by: Cassandra Coyle <[email protected]>

---------

Signed-off-by: Cassandra Coyle <[email protected]>
Co-authored-by: Josh van Leeuwen <[email protected]>

* Workflow: Update durabletask-go to v0.6.1 (#8454)

* Workflow: Adds int tests for ContinueAsNew

Signed-off-by: joshvanl <[email protected]>

* Update tests/integration/suite/daprd/workflow/noawait.go

Co-authored-by: Anton Troshin <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>

* Update tests/integration/suite/daprd/workflow/continueasnew/base.go

Co-authored-by: Anton Troshin <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>

* Update tests/integration/suite/daprd/workflow/continueasnew/activity.go

Co-authored-by: Anton Troshin <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>

* Adds raise continue as new int test

Signed-off-by: joshvanl <[email protected]>

* Raise events in parallel

Signed-off-by: joshvanl <[email protected]>

* Use wrappers int for passing value

Signed-off-by: joshvanl <[email protected]>

* lint

Signed-off-by: joshvanl <[email protected]>

* Updates durabletask to v0.6.1, Adds workflow pause int test

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>
Co-authored-by: Anton Troshin <[email protected]>
Co-authored-by: Elena Kolevska <[email protected]>
Co-authored-by: Cassie Coyle <[email protected]>

* Go: update to v1.23.6 (#8469)

Signed-off-by: joshvanl <[email protected]>

* Scheduler: Enforce 3 replicas on Kubernetes (#8464)

* Scheduler: Update/Remove member hosts

When in Kubernetes mode, Scheduler will update the existing member on
boot, and remove it on shutdown. This is to attempt to prevent member ID
collisions on restarts and upgrades.

Update the client listen address to be only over localhost, removing the
need for mTLS.

Updates dapr/kit to v0.15.0

Signed-off-by: joshvanl <[email protected]>

* Enforce Scheduler to run with 3 replicas always in Kubernetes

Signed-off-by: joshvanl <[email protected]>

* Add fix for service invocation int test

Signed-off-by: joshvanl <[email protected]>

* Return Context to etcd client

Signed-off-by: joshvanl <[email protected]>

* Change embeded etcd into a closer to defer closing etcd till the end of
shutdown

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* fix sidecar injector not respecting annotations set to empty string and a space for placement and scheduler addresses and disable using them accordingly

Signed-off-by: Cassandra Coyle <[email protected]>

* add int test for actors being disabled check

Signed-off-by: Cassandra Coyle <[email protected]>

* Workfkow: Adds integration that an ID cannot be reused (#8471)

* Workfkow: Adds integration that and ID cannot be reused

Signed-off-by: joshvanl <[email protected]>

* AAdds concurrent schedule for workflow ID reuse

Signed-off-by: joshvanl <[email protected]>

* Ensure one error is also nil

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* change unit test bc with that field set to emtpy we default to false, which is same logical behavior if unset bc its false still

Signed-off-by: Cassandra Coyle <[email protected]>

* make lin

Signed-off-by: Cassandra Coyle <[email protected]>

* wfengine holds until ctx.done when actors/placement is disabled

Signed-off-by: Cassandra Coyle <[email protected]>

* rm extra diff

Signed-off-by: Cassandra Coyle <[email protected]>

* make lint

Signed-off-by: Cassandra Coyle <[email protected]>

* dapr/durabletask-go: update to v0.6.2 (#8475)

Signed-off-by: joshvanl <[email protected]>

* Jobs: Adds integration tests for Job loadbalancing

Adds integration test to ensure that jobs are round robin load balanced
across the replicas of an app ID. Developers should never rely
on the durability of Job round robin since the routing is extremely
fragile to connection events, and should therefore be treated as
random.

Signed-off-by: joshvanl <[email protected]>

* Lint

Signed-off-by: joshvanl <[email protected]>

* rm placementenabled check

Signed-off-by: Cassandra Coyle <[email protected]>

* Apply suggestions from code review

Co-authored-by: Josh van Leeuwen <[email protected]>
Signed-off-by: Cassie Coyle <[email protected]>

* fix format from batch

Signed-off-by: Cassandra Coyle <[email protected]>

* rename test struct actors -> emptyaddress

Signed-off-by: Cassandra Coyle <[email protected]>

* fix lint

Signed-off-by: Cassandra Coyle <[email protected]>

* update the helm chart readme to have full set of scheduler flags available (#8478)

Signed-off-by: Cassandra Coyle <[email protected]>

* Integration: Increase timeout for http state error metric (#8463)

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Elena Kolevska <[email protected]>
Co-authored-by: Cassie Coyle <[email protected]>

* Adds workflow tests with durable task client

Signed-off-by: Elena Kolevska <[email protected]>

* Linter

Signed-off-by: Elena Kolevska <[email protected]>

* Scheduler: Cleanup data dir on first boot (#8480)

* Scheduler: Manually add members to Etcd cluster in Kubernetes mode

To avoid member peer id reuse errors, as well as peer URLs not being
correctly updated between upgrades, add a manual peer member process to
Scheduler on startup.

The dapr-scheduler-server-0 replica on boot will check whether it needs
to join members to the cluster from a previous install. It does this by
checking for the existence of the following two files which are written
when adding a member is successful.

```go
filepath.Join(opts.DataDir, instance0Name+"-existing-cluster-1")
filepath.Join(opts.DataDir, instance0Name+"-existing-cluster-2")
```

If these files don't exist, replica 0 will start etcd in single member
mode, to add each member URL manually. Starting the single member ETCD
cluster is gated on the other scheduler replicas making a call to the
gRPC `Instance0Ready` endpoint to signal that they are waiting.  Replica
0 returns from this endpoint for each replica after adding them.

Signed-off-by: joshvanl <[email protected]>

* Instead of membership management shenanigans, delete  data dir on first
boot

Signed-off-by: joshvanl <[email protected]>

* lint

Signed-off-by: joshvanl <[email protected]>

* Fix delete dir error handling

Signed-off-by: joshvanl <[email protected]>

* Make DNS for certificate SAN a wildcard

Signed-off-by: joshvanl <[email protected]>

* Fix scheduler peer cert DNS check

Signed-off-by: joshvanl <[email protected]>

* Fix unit tests

Signed-off-by: joshvanl <[email protected]>

* Update etcd.go

Co-authored-by: Cassie Coyle <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>

* Update etcd.go

Co-authored-by: Cassie Coyle <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>

* Remove the use of wildcard DNS in scheduler certs

Signed-off-by: joshvanl <[email protected]>

* Use scheduler ID on framework scheduler ID

Signed-off-by: joshvanl <[email protected]>

* Fix scheduler cluster framework ID

Signed-off-by: joshvanl <[email protected]>

* Fix int test & data rm on boot

Signed-off-by: joshvanl <[email protected]>

* Set scheduler id in int test

Signed-off-by: joshvanl <[email protected]>

* Set http scheme in scheduler in selfhosted if mtls enabled

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>
Co-authored-by: Cassie Coyle <[email protected]>

* Bumps durabletask-go

Signed-off-by: Elena Kolevska <[email protected]>

* remove repeated tests

Signed-off-by: Elena Kolevska <[email protected]>

* Simplifies test

Signed-off-by: Elena Kolevska <[email protected]>

* Linter

Signed-off-by: Elena Kolevska <[email protected]>

* Typo

Signed-off-by: Elena Kolevska <[email protected]>

* Stricter check

Signed-off-by: Elena Kolevska <[email protected]>

* rename file to emptyaddr

Signed-off-by: Cassandra Coyle <[email protected]>

* require -> assert

Signed-off-by: Cassandra Coyle <[email protected]>

* mv metrics pkg

Signed-off-by: Cassandra Coyle <[email protected]>

* Issue 8254: Added support for setting allowPrivilegeEscalation and readOnlyRootFilesystem as applicable (#8435)

* Issue 8254: Added support for setting allowPrivilegeEscalation and readOnlyRootFilesystem as applicable

Signed-off-by: mustaFAB53 <[email protected]>

* Issue 8254: Made suggested changes

Signed-off-by: mustaFAB53 <[email protected]>

* Issue 8254: Made minor change in default value for dapr scheduler

Signed-off-by: mustaFAB53 <[email protected]>

* Issue 8254: Added README changes as needed

Signed-off-by: mustaFAB53 <[email protected]>

* Issue 8254: Hardcoding readOnlyRootFilesystem to true in scheduler statefulset template as suggested

Signed-off-by: mustaFAB53 <[email protected]>

* Issue 8254: Hardcoded allowPrivilegeEscalation to false in templates as suggested

Signed-off-by: mustaFAB53 <[email protected]>

* Set readOnlyRootFilesystem to false on scheduler

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: mustaFAB53 <[email protected]>
Signed-off-by: joshvanl <[email protected]>
Co-authored-by: joshvanl <[email protected]>
Co-authored-by: Elena Kolevska <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* Add configurable etcd performance optimization settings for Scheduler (#8482)

* add in additional etcd flags for performance optimizations

Signed-off-by: Cassandra Coyle <[email protected]>

* fix test failures

Signed-off-by: Cassandra Coyle <[email protected]>

* httpendpoint test fix

Signed-off-by: Cassandra Coyle <[email protected]>

* require -> assert metrics

Signed-off-by: Cassandra Coyle <[email protected]>

* rm metrics changes

Signed-off-by: Cassandra Coyle <[email protected]>

* snapshotcount set back to 10k, maxwals & snapshots 10 per @yaron2 suggestion

Signed-off-by: Cassandra Coyle <[email protected]>

* lower heartbeat & election interval bc I believe its allowing for extra triggers due to longer transition periods

Signed-off-by: Cassandra Coyle <[email protected]>

* rm configurability for election & heartbeat interval for functional correctness

Signed-off-by: Cassandra Coyle <[email protected]>

* try electiontickadvance false

Signed-off-by: Cassandra Coyle <[email protected]>

---------

Signed-off-by: Cassandra Coyle <[email protected]>
Co-authored-by: Josh van Leeuwen <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* [1.15] update contrib to 1.15.0-rc.2 (#8486)

* [1.15] update contrib to 1.15.0-rc.2

Signed-off-by: yaron2 <[email protected]>

* update dep

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>

* make lint

Signed-off-by: Cassandra Coyle <[email protected]>

* assert instead of require and use collect c instead of parent

Signed-off-by: Cassandra Coyle <[email protected]>

* wrap all metrics in assert eventually, pass in child c instead of parent t

Signed-off-by: Cassandra Coyle <[email protected]>

* cleanup/run once

Signed-off-by: Cassandra Coyle <[email protected]>

* cleanup state between go test count runs and pass listen address thru for operator,dapr,place with @joshvanl

Signed-off-by: Cassandra Coyle <[email protected]>

* run/cleanup once

Signed-off-by: Cassandra Coyle <[email protected]>

* fix CI

Signed-off-by: Cassandra Coyle <[email protected]>

* diagrid.io/go-etcd-cron: update to v0.4.2 (#8489)

Also sneaks in a log fmt fix.

Signed-off-by: joshvanl <[email protected]>

* Fix dapr-injector RoleBinding missing namespace in chart (#8490)

Signed-off-by: Anton Troshin <[email protected]>

* Scheduler: version the Scheduler data dir (#8491)

* Scheduler: version the Scheduler data dir

Version the data dir used by Scheduler Etcd using a new directory
(`dapr-0.1`) so that downgrading to 1.14 does not use the same existing
metadata. Without this, 1.14 which doesn't use TLS will never achieve
quorum and fail to start.

Signed-off-by: joshvanl <[email protected]>

* Fixes unit test checking data dir writeable path

Signed-off-by: joshvanl <[email protected]>

* Fix filepath check for windows

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* add check on if placement is disabled to not start wf engine

Signed-off-by: Cassandra Coyle <[email protected]>

* Actors: Add Placement client reconnect interval (#8493)

* Actors: Add Placement client reconnect interval

Adds a 1s duration interval before attempting to reconnect to placement
in the event of a disconnect, likely due to leadership changes. This
prevents large log spam due to a tight loop in the reconnect logic,
where it will likely take some time for the new placement leader to be
elected.

Signed-off-by: joshvanl <[email protected]>

* Int test: if assert fails in int, return nil to prevent panic

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* Actors: Fix Placement client reconnect (#8497)

* Actors: Fix Placement client reconnect

Fixes an issue whereby a deadlock would occur when halting all actors,
which happens during shutdown or placement dissemination events. This
would happen more often during workflow execution as workflow and
activity actors have a short deactivation time, and the bug involved a
race condition of ranging over the actor table and deleting elements
from the table.

Fixes the actor placement client reconnect logic to be more robust by
cancelling a placement connection when either a `Send` or `Recv` fails.
Also dereferences the placement gRPC connection in the event of
connection refused errors to ensure that the round robin load balance
state is reset so new placement IPs can be discovered from the headless
service. These errors occur when 1 or more placement pods are completely
deleted.

Suppresses an EOF error when streaming workflow runtime status, as this
is a normal condition when the workflow actor stream has been
terminated.

Signed-off-by: joshvanl <[email protected]>

* Remove duplicate Placement connect log line

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* switch from runtimestatus to err check on actors.engine

Signed-off-by: Cassandra Coyle <[email protected]>

* Integration: Prevent panic in metric integration tests (#8498)

* Integration: Prevent panic in metric integration tests

Signed-off-by: joshvanl <[email protected]>

* Fix panic in workflow check state

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>
Co-authored-by: Cassie Coyle <[email protected]>

* diagridio/go-etcd-cron: v0.4.3 (#8500)

* diagridio/go-etcd-cron: v0.4.3

Signed-off-by: joshvanl <[email protected]>

* Remove healthz not ready to prevent killing daprd during dissemination

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* Fix scheduler/placement empty address handling to prevent errors (#8502)

* update sicecar setFromAnnotations func to trim extra double and single quotes to pipe thru the host addresses properly and not result in error if wrapped quotes

Signed-off-by: Cassandra Coyle <[email protected]>

* add placement address check

Signed-off-by: Cassandra Coyle <[email protected]>

* add scheduler test

Signed-off-by: Cassandra Coyle <[email protected]>

---------

Signed-off-by: Cassandra Coyle <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* add placement to int test

Signed-off-by: Cassandra Coyle <[email protected]>

* [1.15] Conversation: change property name from message to content (#8510)

* conversation: change property name from message to content

Signed-off-by: yaron2 <[email protected]>

* update test

Signed-off-by: yaron2 <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>

* tests: add placement process options

Signed-off-by: Fabian Martinez <[email protected]>

* processor: fix waitgroup panic (#8307)

panic: sync: WaitGroup is reused before previous Wait has returned.
Backported from https://github.com/dapr/dapr/pull/8307

Signed-off-by: Luis Rascao <[email protected]>

* add config option for workflows event sink (#8506)

Signed-off-by: Fabian Martinez <[email protected]>

* add extra etcd flags to docs (#8516)

Signed-off-by: Cassandra Coyle <[email protected]>

* ct instead of t (#8515)

Signed-off-by: Cassandra Coyle <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>

* ci: remove dependencies from build job

Signed-off-by: Mike Nguyen <[email protected]>

* add metrics/tracing for conversation api

Signed-off-by: yaron2 <[email protected]>

* Fix Actor & Scheduler deadlocks and Integration Test fixes (#8518)

* Fix Actor & Scheduler deadlocks and Integration Test fixes

This changes includes a number of changes to prevent dead locks in the
actor runtime & Scheduler connectivity/job triggering sequences. There a
number of disparate scenarios covered below. This change ensures that no
deadlocks occur, particularly when running workflows, during scale (up
or down) events, loosing connection or destruction of the placement
and scheduler control plane services. Makes integration tests more
stable, particularly in CI/slow machine runners.

Scenarios and fixes:

Adds a new `scheduler` fields to the gRPC and HTTP metadata output,
which includes an object containing the list of addresses which the
daprd is currently connected to. This address list can change over time,
as daprd dynamically connects/looses connection to scheduler hosts. Int
tests added for this field, as well as being used to check the address
list in some integration tests where the number of hosts connected is a
dependency of that test scenario (e.g. job load balancing).

Moves the placement lock to `dapr/kit/concurrency/lock` as a new shared
`OuterCancel` lib. This struct was also fixed so that a deadlock could
not occur on `RLock` whereby the given context was cancelled, but
`RLock` did not return with error- now it does. This caused deadlocks in
workflow actors whereby dissemination (deactivation) of actors would
not complete as their was a circular lock which did not break.

Removes a deadlock in the Scheduler server whereby a new daprd stream
was stuck adding to the connection pool. This was due to adding the new
connections as deliverable prefixes, locks the underlying go-etcd-cron
queue for those prefixes. If a job trigger was occurring at the same
time, there was a circular lock with no break since the same Scheduler
lock adding the new connection was RLocked to get the connection for
that job trigger to send the job event to daprd. The use of the new
`Context` lock from `dapr/concurrency/lock` ensures that getting the
connection for a triggered job will fail the RLock if it does not gain
the lock after 250ms. This breaks the circular lock. When this RLock
fails, the job will be reported back to go-etcd-cron that the job is not
deliverable and unlock the queue lock- since the job _is_ actually
deliverable, go-etcd-cron will correctly re-enqueue the job immediately
whilst releasing that queue lock in the mean time. See:
https://github.com/diagridio/go-etcd-cron/blob/c9126ef694ffed83011125c5acbf6127f33bf784/internal/queue/queue.go#L209-L217

Fixes another deadlock in the Scheduler server whereby a connection to
daprd would not correctly be identified as a closed or cancelled
connection. This would happen when the connection is closed or
cancelled, whilst there is a send & job handle in progress for a
particular connection. This has been resolved by adding another go
routine to the connection handler to detect when either Recv or Send has
returned, the pool total is closed, or the stream context has closed,
then closing and removing that connection entirely. This correctly
ensures that the connection is removed from the pool, as well as
returning from the stream at the top gRPC RPC level.

Permanently backoff when calling a remote reminder when the remote peer
is not available, i.e. during a scale down event. This prevents future
retries, which slows down the entire placement dissemination for that
namespace.

Mark the placement client as ready later- after the placement client
has sent the first host request, so to better reflect the real ready
state and prevent unnecessary timing errors which cause retries
downstream.

Move `HaltAll` and `Drain` implementation to the same func as they were
the same implementation, only with one using a conditional for the
drain.

Pre-fetch the actor sub Interfaces on the Factory initialization of
workflow and activity actors. This prevents a circular deadlock during
cases of dissemination where a workflow/activity actor would be
deactivated whilst attempting to retrieve one of these interfaces (like
state).

Move the workflow actor internal lock to only be used in the _actual_
critical shared memory locations. Previous use of locking around actor
resource usage like state would cause the same circular deadlock as
described above.

The Scheudler client has been changed to be more aggressive for
scheduler reconnects, particularly on `WatchHosts`. This prevents
reconnect errors which were observed in scenarios whereby the entire
scheduler cluster was to go down temporarily. Also removes the retry
loop for the Scheduler client connector since a reconnect at the top
level is preferred.

Mark the Scheduler client readiness to true only when all Scheduler
host clients have actually sent the initial message to their respective
host. Sets readiness to false during shutdown and reconnect events.

JobsWatch passes a context up which will be closed when either
Send/Recv or shutdown is in progress to ensure the gRPC API is returned
on disconnect events.

Set Scheduler log level to `debug` in e2e CI runs.

Integration Tests:

Set all exec processes to use `debug` log level to make debugging easier
when Integration tests fail.

Fixes shutdown timeout of HTTP app Cleanup.
Fix log line process Cleanup logic to ensure all exec process logs are
flushed before checking for a log line existence.
Fixes timing errors in scheduler failure policy tests.
Adds actor type handler not implementing the deactivation endpoints.
Reduce the number of components in fuzz component name tests to prevent
shutdown timeouts on slow CI runnners.
Fix broken pipe network errors which occur on slow CI runners.
Adds tests for the new scheduler metadata field, both for gRPC and HTTP.
Fix state metrics tests by using the correct Enventually tester on
asserts.
Correctly use Workflow `WaitForSingleEvent` for workflow coordination,
rather than cross scope channels for singling.
Use eventuallys to prevent EOF errors during placement dissemination
tests.

Signed-off-by: joshvanl <[email protected]>

* Updates dapr/kit to v0.15.1

Signed-off-by: joshvanl <[email protected]>

* Close send job context on close & mark deliverable context after setting
up conn

Signed-off-by: joshvanl <[email protected]>

* Update go-etcd-cron to use joshvanl fork for context cancelling
DeliverablePrefixes

Signed-off-by: joshvanl <[email protected]>

* Move connection context manager into pool with removal handler for
DeliverablePrefixes

Signed-off-by: joshvanl <[email protected]>

* Only send actor types to Scheduler if using Scheduler reminders

Signed-off-by: joshvanl <[email protected]>

* Move custom `proto` grpc codec registry to runtime init

Signed-off-by: joshvanl <[email protected]>

* int: Fix actor remote http call channel call race

Signed-off-by: joshvanl <[email protected]>

* empty

Signed-off-by: joshvanl <[email protected]>

* Use correct feature config for actor reminder partition based test

Signed-off-by: joshvanl <[email protected]>

* Set http client response header timeout to 5s

Signed-off-by: joshvanl <[email protected]>

* Actors: don't lock on table list types

Signed-off-by: joshvanl <[email protected]>

* Add Eventually around component update scopes

Signed-off-by: joshvanl <[email protected]>

* Increase http response headers to 8s

Signed-off-by: joshvanl <[email protected]>

* Increase header response timeout to 15s

Signed-off-by: joshvanl <[email protected]>

* diagridio/go-etcd-cron: update to v0.4.4

Signed-off-by: joshvanl <[email protected]>

* empty

Signed-off-by: joshvanl <[email protected]>

* empty

Signed-off-by: joshvanl <[email protected]>

* Do not gracefully drain actors when calling table `HaltAll`. Adds
`Type()` and `ID()` to actor targets Interface & cleanup actor table
funcs logic.

Signed-off-by: joshvanl <[email protected]>

* Add backoff to getRemindersForActorType to account for etag missmatch
race

Signed-off-by: joshvanl <[email protected]>

* Prevent exec error panicing whole suite & include timeout before closing
exec std pipes

Signed-off-by: joshvanl <[email protected]>

* e2e/scheduler_test: remove time.Sleep

Signed-off-by: joshvanl <[email protected]>

* Increase timeout for waiting for actor hop finish

Signed-off-by: joshvanl <[email protected]>

* Reduce number of partitions to test to 4

Signed-off-by: joshvanl <[email protected]>

* Increase individual e2e test run to 20m

Signed-off-by: joshvanl <[email protected]>

* Make getStream accept Eventually collect tester

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* update contrib to rc.4

Signed-off-by: yaron2 <[email protected]>

* linter

Signed-off-by: yaron2 <[email protected]>

* upgrade contrib to rc.4

Signed-off-by: yaron2 <[email protected]>

* fix workflow name empty on first event

Signed-off-by: Fabian Martinez <[email protected]>

* golang.org/x/crypto: update to v0.35.0 (#8528)

* golang.org/x/crypto: update to v0.35.0

Signed-off-by: joshvanl <[email protected]>

* mod tidy all

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* fix workflows closing broadcaster subscriptions (#8525)

Signed-off-by: Fabian Martinez <[email protected]>
Co-authored-by: Josh van Leeuwen <[email protected]>

* second attempt of unmarshalling should be to json (#8527)

unmarshal to string

Signed-off-by: Elena Kolevska <[email protected]>
Co-authored-by: Josh van Leeuwen <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>

* update contrib to latest (#8533)

Signed-off-by: yaron2 <[email protected]>

* new PR (#8536)

Signed-off-by: Cassandra Coyle <[email protected]>

* Release notes - 1.15 (#8466)

* release: add notes

Signed-off-by: Mike Nguyen <[email protected]>

* release: update rust sdk notes and add dapr workflow method deprecation notice

Signed-off-by: Mike Nguyen <[email protected]>

* release: update componentscontrib and go-sdk notes

Signed-off-by: Mike Nguyen <[email protected]>

* release: add rust highlight

Signed-off-by: Mike Nguyen <[email protected]>

* release: bump notes

Signed-off-by: mikeee <[email protected]>

* release: bump

Signed-off-by: Mike Nguyen <[email protected]>

* release: bump relnotes

Signed-off-by: Mike Nguyen <[email protected]>

* release: bump notes

Signed-off-by: Mike Nguyen <[email protected]>

* release: bump notes

Signed-off-by: Mike Nguyen <[email protected]>

* release: bump docs

Signed-off-by: Mike Nguyen <[email protected]>

* release: docs bump

Signed-off-by: Mike Nguyen <[email protected]>

* release: update docs

Signed-off-by: Mike Nguyen <[email protected]>

* release: fix links

Signed-off-by: Mike Nguyen <[email protected]>

---------

Signed-off-by: Mike Nguyen <[email protected]>
Signed-off-by: mikeee <[email protected]>

* [1.15] Add missing grpc call options when invoking actors (#8541)

* add missing grpc call options when invoking actors

Signed-off-by: yaron2 <[email protected]>

* adds release notes + changes options struct

Signed-off-by: yaron2 <[email protected]>

* remove redundant struct

Signed-off-by: yaron2 <[email protected]>

* Adds integration test for max-body-size for actor invocation

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: yaron2 <[email protected]>
Signed-off-by: joshvanl <[email protected]>
Co-authored-by: joshvanl <[email protected]>

* [v1.15] Actors: Don't sent Reentrancy ID header when disabled (#8542)

* [v1.15] Actors: Don't sent Reentrancy ID header when disabled

Fix sending Actor Reentrancy ID HTTP header when disabled

Problem
Calling `this.StateManager.TryGetStateAsync()` in the DotNet Actor SDK would return stale data during some invocation scenarios.

Impact
The latest Actor state data was not being correctly returned during some Actor invocation scenarios using the DotNet Actor SDK.

Root cause
When Reentrancy was disabled, the Actor Reentrancy ID HTTP header was still being sent to Actor HTTP servers.
The DotNet SDK uses the existence of this HTTP header in logic to determine what state should be returned to the Actor.

Solution
Don't send the Actor Reentrancy ID HTTP header (`"Dapr-Reentrancy-Id"`) when Reentrancy is disabled.

Signed-off-by: joshvanl <[email protected]>

* Fix actor call placement lock due to reentrancy ID HTTP header.

Remove reentancy ID HTTP header check from placement lock during actor
calls. Since we always added the reentrancy ID header in v1.15.0,
`isReentrancy` bool was always true, so `!isReentrancy` was always
false- a no-op to the OR statement. This was added before other changes
were made to the actor locking system, but since it was a no-op, it was
never caught and removed. Now the reentrancy ID header is (correctly)
optionally added to keep backwards compatibility, and the previous actor
engine state this check was catering for no longer exists, this check
actually breaks the runtime.

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* [v1.15] Actors: Always attempt to connect to placement if configured. (#8546)

* [v1.15] Actors: Always attempt to connect to placement if configured.

Fixes a regression whereby daprd would no longer connect to placement if
an Actor state store is not configured. This meant an app which did not
have an actor state store configured (& host Actors), could not invoke
Actors on other apps.

Update daprd to always start Placement and the Actor runtime if
Placement addresses are configured.

Adds integration test for this scenario.

Adds metadatda integration tests.

Signed-off-by: joshvanl <[email protected]>

* Move actor store registration on state store existing

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: joshvanl <[email protected]>

* [1.15] Scheduler: Fix go routine memory leak (#8547)

This update includes bug fixes:

- [Fix Scheduler memory leak](#fix-scheduler-memory-leak)

Fix Scheduler memory leak

Problem

Scheduler would continually grow in memory during usage.

Impact

Scheduler would consume the entirety of memory on the host machine, or to it's cgroup limit, then OOM crash.

Root cause

A Go routine was doing being closed on connection sends.

Solution

Correctly close the Go routine on completion of the connection send.

Signed-off-by: joshvanl <[email protected]>

* [v1.15] Actors: Always tick timer invocations forwards. (#8549)

Dapr 1.15.3

This update includes bug fixes:

- [Fix Timers Deactivating after timer invocation fails](#fix-timers-deactivating-after-timer-invocation-fails)

Fix Timers Deactivating after timer invocation fails

Problem

An app returning a non-2xx status code from a timer invocation would cause a periodic timer to no longer trigger.

Impact

An Actor app which restarted/crashed, or was otherwise busy, would cause a timer to no longer trigger.
This breaks backwards compatibility where a periodic Actor timer would continue to trigger at the defined period, even if the actor was busy or had an error.

Root cause

The Actor timer handle logic deactivates the timer if _any_ timer invocation failed.
Regardless of whether the timer had further ticks defined in it's period schedule.

Solution

As did before v1.15.0, treat any successful or failed timer invocation as the same, and tick the Actor timer forward allowing for future invocations.

Signed-off-by: joshvanl <[email protected]>

---------

Signed-off-by: Elena Kolevska <[email protected]>
Signed-off-by: Luis Rascao <[email protected]>
Signed-off-by: joshvanl <[email protected]>
Signed-off-by: Cassandra Coyle <[email protected]>
Signed-off-by: Cassie Coyle <[email protected]>
Signed-off-by: Anton Troshin <[email protected]>
Signed-off-by: Josh van Leeuwen <[email protected]>
Signed-off-by: mustaFAB53 <[email protected]>
Signed-off-by: yaron2 <[email protected]>
Signed-off-by: Fabian Martinez <[email protected]>
Signed-off-by: Mike Nguyen <[email protected]>
Signed-off-by: mikeee <[email protected]>
Co-authored-by: Elena Kolevska <[email protected]>
Co-authored-by: Luis Rascao <[email protected]>
Co-authored-by: Yaron Schneider <[email protected]>
Co-authored-by: Elena Kolevska <[email protected]>
Co-authored-by: Dapr Bot <[email protected]>
Co-authored-by: Cassie Coyle <[email protected]>
Co-authored-by: Anton Troshin <[email protected]>
Co-authored-by: Loong Dai <[email protected]>
Co-authored-by: Anton Troshin <[email protected]>
Co-authored-by: Cassie Coyle <[email protected]>
Co-authored-by: Mustafa Bhabhrawala <[email protected]>
Co-authored-by: Fabian Martinez <[email protected]>
Co-authored-by: Mike Nguyen <[email protected]>