[QUERY] Application took 15 minutes to update checkpoint

[neallee2012]

Describe the bug

Receive event: "12/7/2020, 6:21:43.610"
Finish processing event: "12/7/2020, 6:21:43.669"
Update Checkpoint: "12/7/2020, 6:37:26.121"

Checkpoint writing stucked in application for 15minutes. There is a duration of ownership expiration set as partitionOwnershipExpirationInterval(Duration.ofSeconds(150)) during creation.

To Reproduce

Keep updating checkpoint and It will happen occasionally

Code Snippet

public final Consumer<EventContext> PARTITION_PROCESSOR = eventContext -> {
        Map<String, Object> metricMap = new HashMap<>();

        EventData event = eventContext.getEventData();
        metricMap.put(OWNER_ID, ownerId);
        metricMap.put(OFFSET, event.getOffset());
        metricMap.put(SEQUENCE_NUMBER, event.getSequenceNumber());
        metricMap.put(ENQUEUE_TIME, event.getEnqueuedTime().getEpochSecond());
        metricMap.put(PARTITION_ID, eventContext.getPartitionContext().getPartitionId());
        metricMap.put(EVENT_ID, String.format("%s_%s_%d", metricMap.get(OWNER_ID), metricMap.get(PARTITION_ID), (long) metricMap.get(SEQUENCE_NUMBER)));

        String receivedEventContext = metricMap.entrySet().stream().map((entry) -> (entry.getKey() + "=" + entry.getValue().toString())).sorted().collect(Collectors.joining(" & "));
        LOG.info("receive event: {}={}", EVENT_ID, metricMap.get(EVENT_ID));

        metricMap.put(EVENT_HUB_SINK_COUNT, "0");
        metricMap.put(EVENT_HUB_SINK_TIME, "0");
        metricMap.put(BLOB_STORAGE_SINK_SIZE, "0");
        metricMap.put(BLOB_STORAGE_SINK_TIME, "0");
        metricMap.put(TRANSFORMATION_TIME, "0");

        long lag = Instant.now().getEpochSecond() - (long) metricMap.get(ENQUEUE_TIME);

        statsd.recordGaugeValue("xxxxxxx", lag, "partition_id:" + metricMap.get(PARTITION_ID));

        metricMap.put(EXECUTION_TIME, getExecutionTime(() -> processEvent(event.getBody(), metricMap)));

        String executionDetails = metricMap.entrySet().stream().map((entry) -> (entry.getKey() + "=" + entry.getValue().toString())).sorted().collect(Collectors.joining(" & "));
        LOG.info("finish event processing: " + executionDetails);

        eventContext.updateCheckpoint();
        LOG.info("update checkpoint: {}={}", EVENT_ID, metricMap.get(EVENT_ID));
    };

Expected behavior

It should either fail or succeed in shorter time.

Setup (please complete the following information):

• OS: Linux
• Version of the Library used: 5.3.1

[YijunXieMS]

Hi @alzimmermsft,
The updateCheckpoint in EventHubs blob checkpoint store calls the blob storage API blobAsyncClient.setMetadata(metadata);
Among many calls, one call was stuck for 15 minutes and then succeeded. Other calls about the same time in different threads succeeded immediately. Do you know of any similar cases?

Another question is, can we configure the timeout for the above API? If it times out in much shorter time like in a few seconds, the updateCheckpoint won't be blocking for 15 minutes. An error is much better than a 15-minute waiting.

[alzimmermsft]

Based on the dependencies of azure-messaging-eventhubs-checkpointstore-blob it resolves to azure-core-http-netty 1.5.4, in a new release of this library (1.6.0 and above) additional logic was added to timeout requests/responses after a certain time of inactivity. Upgrading to a new version of azure-storage-blob or azure-core-http-netty should add in this functionality which should help reduce the chances of hitting a 15 minute wait period in the application.

[YijunXieMS]

@neallee2012 Following Alan's answer above, could you put azure-storage-blob 12.9.0 in your pom.xml before we release a new checkpoint store blob version that uses storage 12.9.0?

<!-- https://mvnrepository.com/artifact/com.azure/azure-storage-blob -->
<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-storage-blob</artifactId>
    <version>12.9.0</version>
</dependency>

[srnagar]

@neallee2012 Do you have any updates? Did you try the new storage blob version?

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!