Checklist
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Description
First off, documentation is not especially wrong on the use of DynamoDB as a results backend. However, it is incomplete in a significant way. Because I manage the DynamoDB table through IaC I did not realize that defining a sort key would break compatibilty with Celery.
When you use a DynamoDB table with a sort key you can not use Celery v4.4 versions >= v4.4.3. The reason is commit c711047 which introduces a query of existing Celery result status. When you have a sort key defined for your DynamoDB table, the primary key is no longer just the partition key ('id'), but a combination of the partition and sort keys ('id' and 'timestamp' in my case).
Suggestions
It would be fun to add sort key support to the DynamoDB results backend, but I fear it will not be possible. For example, in my case I will need to know the value of the 'timestamp' in addition to the 'id' of the result item we need to retrieve. I don't believe Celery has knowledge of the 'timestamp' we are looking for at the time the query from commit c711047 is performed.
So my suggestion is simple: Let's add a line in the documentation about DynamoDB use as results backend that warns us that you cannot use a sort key on the table. Luckily, you could still add a global secondary index (GSI) if you want to perform queries based on something other than the partition key. A GSI will not break the primary key schema for the queries from Celery.
Checklist
for similar or identical bug reports.
for existing proposed fixes.
to find out if the bug was already fixed in the master branch.
(If there are none, check this box anyway).
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Description
First off, documentation is not especially wrong on the use of DynamoDB as a results backend. However, it is incomplete in a significant way. Because I manage the DynamoDB table through IaC I did not realize that defining a sort key would break compatibilty with Celery.
When you use a DynamoDB table with a sort key you can not use Celery v4.4 versions >= v4.4.3. The reason is commit c711047 which introduces a query of existing Celery result status. When you have a sort key defined for your DynamoDB table, the primary key is no longer just the partition key ('id'), but a combination of the partition and sort keys ('id' and 'timestamp' in my case).
Suggestions
It would be fun to add sort key support to the DynamoDB results backend, but I fear it will not be possible. For example, in my case I will need to know the value of the 'timestamp' in addition to the 'id' of the result item we need to retrieve. I don't believe Celery has knowledge of the 'timestamp' we are looking for at the time the query from commit c711047 is performed.
So my suggestion is simple: Let's add a line in the documentation about DynamoDB use as results backend that warns us that you cannot use a sort key on the table. Luckily, you could still add a global secondary index (GSI) if you want to perform queries based on something other than the partition key. A GSI will not break the primary key schema for the queries from Celery.