description

After upgrading an existing fdp-index from v1.17 (or older) to v1.18.0, the list of FDPs is empty.

Inspection of the mongodb content shows that the indexEntry collection still contains the existing FDP entries.
However, these do not have the additional permit field that was added in v1.18.0.

It is likely that this causes a problem when mapping to the IndexEntry entity or when filtering these entries.

how to reproduce

1. Clone the FAIRDataTeam/compose repo
2. Create a temporary subfolder compose/fdp/custom/v1/index with the following files:
   compose.yml

   include:
     - ../../../components/db/mongo.yml
     - ../../../components/v1/fdp-index.yml
     # this fdp is only here to ping the index, so we get an actual indexEntry
     - ../../../components/v1/fdp.yml

   compose.override.yml

   services:
     fdp-index:
       # set the peristent url to match the exposed port, for consistency
       # (otherwise rdf subject and swagger server do not match actual origin)
       environment:
         INSTANCE_PERSISTENTURL: http://localhost
       # expose the index so we can use the api
       ports:
         - "127.0.0.1:80:8082"
     fdp:
       # make sure FDP_VERSION only changes the index version
       image: fairdata/fairdatapoint:1.17
       environment:
         # use fdp instead of localhost, to bypass the index deny list
         INSTANCE_CLIENTURL: http://fdp:8080
         INSTANCE_PERSISTENTURL: http://fdp:8080
         INSTANCE_INDEX: false
         PING_ENABLED: ${PING_ENABLED:-true}
         # the fdp-index component uses port 8082 by default
         PING_ENDPOINTS: http://fdp-index:8082
         PING_INTERVAL: 3600000  # ms
         # the mongo db name must be different from that of the index (which is "fdp-index")
         SPRING_DATA_MONGODB_URI: mongodb://mongo:27017/fdp-ping
       depends_on:
         # only start pinging when index is ready
         fdp-index:
           condition: service_healthy
     # use docker managed volumes for data persistence
     mongo:
       volumes:
         - mongodbdata:/data/db
   
   volumes:
     mongodbdata:

3. Set up the stack with a v1.17 fdp index, and wait for it to become healthy:

   FDP_VERSION=1.17 docker compose up -d

4. Use the API to verify that an entry is present for http://fdp:8080 (may take some time for the fdp container to send a ping):

   curl http://localhost/index/entries

   expected response is something like {"content":[{"uuid":"...","clientUrl":"http://fdp:8080", ...}], ...}
5. Tear down the fdp-index while keeping the mongo container running:

   docker compose down fdp-index

6. Clear the rdf migration data from the mongodb, as a workaround for Recurring error "No metadata found for the uri '<client uri>'"  #252

   docker compose exec mongo mongosh \
       --eval "use fdp-index" \
       --eval "db.rdfMigration.deleteMany({})" \
       --eval "db.metadata.deleteMany({})"

7. Perform the upgrade to v1.18:

   FDP_VERSION=1.18.0 docker compose up -d fdp-index

8. Use the API again to verify that the list is now empty:

   curl http://localhost/index/entries?permit=ALL

   which returns something like {"content":[], ...}
   Be aware that unauthenticated users can only see fdps with permit: "ACCEPTED".
9. Verify that the indexEntry document does still exist, but does not have a permit field:

   docker compose exec mongo mongosh --eval "use fdp-index" --eval "db.indexEntry.find()"

workaround

A workaround is to add the permit field to existing indexEntry documents manually.

docker compose exec mongo mongosh \
    --eval "use fdp-index" \
    --eval 'db.indexEntry.updateMany({},{$set: {permit: "ACCEPTED"}})'

verify that this works by using the API again:

curl http://localhost/index/entries?permit=ALL

proposed fix

There are at least two ways to fix this properly:

• create a data migration to add permit: "ACCEPTED" to all existing indexEntry documents
• or, modify the entity mapping (or filter, depending on the actual cause) so that it does not fail when fields are missing