Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-36618

Improve PostgresDialect.discoverDataCollections to reduce the start time of Postgres CDC

    XMLWordPrintableJSON

Details

    Description

      Current, PostgresDialect.discoverDataCollections will lookup tableSchemas for each table.

       

      @Override
      public Map<TableId, TableChange> discoverDataCollectionSchemas(JdbcSourceConfig sourceConfig) {
          final List<TableId> capturedTableIds = discoverDataCollections(sourceConfig);
      
          try (JdbcConnection jdbc = openJdbcConnection(sourceConfig)) {
              // fetch table schemas
              Map<TableId, TableChange> tableSchemas = new HashMap<>();
              for (TableId tableId : capturedTableIds) {
                  TableChange tableSchema = queryTableSchema(jdbc, tableId);
                  tableSchemas.put(tableId, tableSchema);
              }
              return tableSchemas;
          } catch (Exception e) {
              throw new FlinkRuntimeException(
                      "Error to discover table schemas: " + e.getMessage(), e);
          }
      }
       

      I have a job with table name which match hundreds of table will start for 1 hour.

       

      Attachments

        Issue Links

          Activity

            People

              loserwang1024 Hongshun Wang
              loserwang1024 Hongshun Wang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: