htwkmooc/poseidon
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix missing rescheduled idle runners.

Browse Source 
In today's unattended upgrade, we have seen how the prewarming pool size dropped to (near) zero. This was based on lost Nomad allocations. The allocations got rescheduled, but not added again to Poseidon.

The reason for this is a miscommunication between the Event Handling and the Nomad Manager. `removedByPoseidon` was true even if the runner was not removed by the manager, but an idle runner.
This commit is contained in:
Maximilian Paß
2023-08-29 23:27:38 +02:00
committed by Sebastian Serth
parent 67297ec5a2
commit 354c16cc37
7 changed files with 46 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
internal/nomad/nomad.go
View File

@ -47,6 +47,9 @@ type AllocationProcessing struct {
OnDeleted DeletedAllocationProcessor
}
type RunnerDeletedReason error
// DeletedAllocationProcessor is a handler that will be called for each deleted allocation.
// removedByPoseidon should be true iff the Nomad Manager has removed the runner before.
type DeletedAllocationProcessor func(jobID string, RunnerDeletedReason error) (removedByPoseidon bool)
type NewAllocationProcessor func(*nomadApi.Allocation, time.Duration)