Skip to content

Conversation

@johha
Copy link
Contributor

@johha johha commented Nov 7, 2025

When creating tasks in parallel, a race condition could lead to quota over-commitment. Tasks are first created in a PENDING state, which was not included in quota calculations. This allowed multiple tasks to be created and submitted to Diego for execution, even if their combined resource usage exceeded the defined quota.
When the task's state was subsequently updated, the model validation would fail due to the exceeded quota, incorrectly marking the task as FAILED. This created a state mismatch, as the task had already been successfully submitted to Diego.

This change resolves the issue by including PENDING tasks in the memory and log rate limit calculations, treating the state as a desired state similar to apps. This prevents the initial over-submission of tasks.

Additionally, the quota validation is now skipped when a task transitions from PENDING to RUNNING. This ensures that a task successfully submitted to Diego is correctly reflected as RUNNING in the Cloud Controller database, aligning the desired state with the action taken.

Fixes #4619

  • I have reviewed the contributing guide

  • I have viewed, signed, and submitted the Contributor License Agreement

  • I have made this pull request to the main branch

  • I have run all the unit tests using bundle exec rake

  • I have run CF Acceptance Tests

Tasks remain in state `PENDING` only for a very short period - from creation until submission to Diego (few ms).
If a user creates many tasks in parallel it might happen that the quota check will fail although the task is already submitted to Diego.
This change adds pending tasks to the remaining memory and log rate limit calculation of spaces and orgs.
In other words pending tasks behave like apps where there is only a desired started and stopped state and no sync with the actual state in Diego.

Additionally the memory and log rate limit quota calculation is now skipped if the task state changes from `PENDING` to `RUNNING` to allow updates even if the quota is exceeded.
@johha johha changed the title Count pending tasks towards memory and log rate limit quota Include pending tasks in memory and log rate limit quota calculations Nov 11, 2025
@johha johha changed the title Include pending tasks in memory and log rate limit quota calculations Include pending tasks in quota calculations Nov 11, 2025
@johha johha marked this pull request as ready for review November 11, 2025 13:47
@johha johha merged commit e0591a9 into main Nov 13, 2025
11 checks passed
@johha johha deleted the pending-tasks-quota branch November 13, 2025 09:37
ari-wg-gitbot added a commit to cloudfoundry/capi-release that referenced this pull request Nov 13, 2025
Changes in cloud_controller_ng:

- Include pending tasks in quota calculations
    PR: cloudfoundry/cloud_controller_ng#4651
    Author: Johannes Haass <johannes.haass@sap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tasks created in parallel might lead to FAILED tasks that are still executed

2 participants