About today's (2023-05-09) github outage :)

Introduction

A colleague asked me if I know what to do about the error with github:

Troubleshooting

At first I was expecting some data conflict or filesystem issue so i recommended 2 options listed bellow and asked about the repo so I can try to reproduce the problem. And I have cloned the repo (just in case using "git+ssh" url instead of https).

`git fsck` checks the internal git filesystem (the git objects).

`git gc` triggers garbage collection. You might be not aware but yes - there is garbage collection even in git! If <something> is in git but is not used (not referenced in any way) then it is kept for 30 days (default value). An example might be that commit you made but then you undo'ed it by moving head (branch) of your code to previous commit. After 30 days have passed the next operation should trigger this cleanup.

Here I should have checked if there are any hooks enabled (but there were not).

Then `git push origin develop` (since that was the branch I was on). This "worked" - I had no changes so nothing was pushed. I had to introduce a dummy change to be pushed.

Dummy commit can be created with `git commit --allow-empty -m "empty commit"` and then I could re-run `git push origin develop`. But I was not allowed to push → the same fail appeared. And it confused me. Could it be some error on the gitgub server?

`git push origin develop -f` (the branch was not protected) should push my repo to github and hopefully override any error that can be there. But I got the error again. First conclusion: this is rather server-side issue than client-side.

Server troubleshooting

I went to the repository web page and its "settings" section. Quick check if the branch is not protected → it was not.

What other reason could be there !? Network issue or temporary github unavailability is not the cause because I can access github and issue can be reproduced by multiple users on multiple computers.

Last idea: collaborators and teams. I was an admin of the repo but not listed in the collaborators. Tried adding myself by user account and.... it worked. No more errors. Could it be that my admin permissions were not correctly used?

Permissions! Let's fix this

I have added my colleague's team to the repository as an admin (I was lucky I could do this without any risk). But then... it was not working! He got the error again.

To my surprise everything worked when I added his user account to the collaborators list. Could it be that teams were broken? (github teams, not the memory hog application called Microsoft Teams which on linux barely works)

What next? We got the cause, we have a dirty workaround. Next step is to inform others. And create a ticket. In some organization ticket would be the first step (you are going to pee →?create a ticket, you are bored →?create a ticket ;) ). Here github showed an outage :O Cool - they already know about it and it is a larger issue. I am not the only one impacted. Possibly this is a significant outage. Might be even something critical. Let's wait and see. No reason to bother Microsoft any more - they are already working on it. In a few days we should hear some feedback (hopefully). The ticket in Microsoft is there.