EphemeralRunner: On cleanup, if pod is pending, delete from service#2255

Merged

TingluoHuang merged 8 commits into

nikola-jokic/manual-delete-issue-delete

Feb 12, 2023

nikola-jokic commented Feb 6, 2023

Collaborator

If pod is unable to start due to image pull issue, the pod will get stuck in Pending phase. It would require manual intervention to delete that ephemeral runner.

This change removes runner from the service if pod is pending (never actually started).

nikola-jokic requested review from a team, mumoshu and toast-gear as code owners

February 6, 2023 15:14

TingluoHuang commented Feb 6, 2023

Member

Will this cause ARC stuck in a loop of creating/deleting runners on the service?

nikola-jokic commented Feb 6, 2023

Collaborator Author

Yes ☺️ . So basically, if the issue is on the container registry, the back-off will eventually pull it.
If the issue is some typo in the image field, you would need to update AutoscalingRunnerSet anyway.
The update will take care of deletion, but if someone manually tries to delete it (maybe by accident or image pull back-off tops at 5 minutes so to speed it up), then the runner will stay registered.

TingluoHuang commented Feb 6, 2023

Member

@nikola-jokic The service will return 429 if you configure too many runners within a short time window.

TingluoHuang reviewed

View reviewed changes

controllers/actions.github.com/ephemeralrunner_controller.go Outdated

nikola-jokic added 2 commits

February 8, 2023 15:23


          EphemeralRunner: On cleanup, if pod is pending, delete from service (…

2cf1224

…ErrImagePull)


          Include finalizer for removing ephemeral runner from the service

b664d11

nikola-jokic force-pushed the nikola-jokic/manual-delete-issue-delete branch from 1c007f4 to b664d11 Compare

February 8, 2023 16:38

TingluoHuang reviewed

View reviewed changes

controllers/actions.github.com/ephemeralrunner_controller.go Outdated

TingluoHuang reviewed

View reviewed changes

controllers/actions.github.com/ephemeralrunner_controller.go Outdated

TingluoHuang reviewed

View reviewed changes

controllers/actions.github.com/ephemeralrunner_controller.go Outdated

nikola-jokic added 2 commits

February 10, 2023 18:14


          Extracted actions cleanup into a method, renamed finalizer

b34c0f7


          fixed reverse logic on error

997d54a

TingluoHuang reviewed

View reviewed changes

controllers/actions.github.com/ephemeralrunner_controller.go Outdated

TingluoHuang reviewed

View reviewed changes

controllers/actions.github.com/ephemeralrunner_controller.go

nikola-jokic and others added 4 commits

February 11, 2023 18:01


          Update controllers/actions.github.com/ephemeralrunner_controller.go

25e5d49

Co-authored-by: Tingluo Huang <tingluohuang@github.com>


          Update controllers/actions.github.com/ephemeralrunner_controller.go

Co-authored-by: Tingluo Huang <tingluohuang@github.com>


          formatted

5b24dcd


          fix suggestion

85f788e

TingluoHuang approved these changes

View reviewed changes

TingluoHuang merged commit 8e52a6d into master

TingluoHuang deleted the nikola-jokic/manual-delete-issue-delete branch

February 12, 2023 00:55

TingluoHuang mentioned this pull request

Bump arc-2 chart version and prepare 0.2.0 release #2313

Merged

unpollito pushed a commit to DistruApp/actions-runner-controller that referenced this pull request


          EphemeralRunner: On cleanup, if pod is pending, delete from service (a…

bc29601

…ctions#2255)

Co-authored-by: Tingluo Huang <tingluohuang@github.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet