Nothing is pissing me off more than GitHub's stability going down the tubes RIGHT as work is migrating everything, and I mean everything, from CircleCI to GH.
The wildest thing is that Azure Repos/Pipelines was better than this.
Their one caveat is also that they are still migrating it to Azure infra, so it's possible that's still in a one foot in one foot out kinda scenario, from what I've heard. But, this isn't inspiring confidence.
Could be. But 99% of the repos are static garbage with no PR nor actions.
They mentioned they have some elasticsearch reindexing going to, I would guess they needed to regard or move stuff and something didn't work well. But if I understood it right they mentioned the PRs ES index which they didn't shared proof increased as the number of repos.
It might be anything. It seems they lost huge chunks due to layoffs and structural changes and MS which has the reverse golden Midas touch.
This is just pure speculation but also now there is no reason for MS to keep GH working. They absorbed all code they wanted. Now they can let it burn. Would be even better for them if that happened
> Could be. But 99% of the repos are static garbage with no PR nor actions.
But the 1% of repos that do have PRs and actions are likely going to be seeing enormous increases in volumes
I have been a part of two very large companies with self hosted gits and I've seen enough to be confident that this is an incredibly hard thing to manage
Serious question, have you been part of an org that had to scale orders of magnitude very quickly?
Anyone who has been part of that journey knows how painful it really is. A lot of times the systems to fail at all levels, and you have to redesign it from the first principles.
They say it is at least one order of magnitude[1]; "our plan to increase GitHub’s capacity by 10X in October 2025 .. By February 2026, it was clear that we needed to design for a future that requires 30X today’s scale."
> Serious question, have you been part of an org that had to scale orders of magnitude very quickly?
I have, but it depends what you mean.
Scenario 1: e-commerce SaaS (think: Amazon but whitelabel, and before CPUs even had AES instructions); Christmas was "fun".
Scenario 2: Video Games. The first day is the worst day when it comes to scale. Everything has to be flawless from day 0 and you get no warning as to what can go wrong.
Yet, somehow, I managed to make highly reliable systems.
In scenario 1; I had an existing system that had to scale up and down with load, this was before there was cloud and hardware had a 3-4 month lead time, so most of the effort was around optimising existing code, increasing job timeouts and "quenching" sources that were expensive. We used to also do so 'magic' when it came to serving requests that had session token or shopping cart cookie.
In scenario 2; we have a clean-room implementation and no legacy, which is a blessing but also a curse, there's no possibility to sample real usage: but you also don't need to worry about making changes that are for the better.
So, pro's and con's... but it's not like handling huge load hasn't been done before, computers are faster than they ever have been and while my personal opinion is that operational knowledge is dying (due to general distain for people who actually used to run systems that scale: not just write hopeful "eventually consistent" yaml that they call deterministic) - the systems that do exist today hold your hand much better than they did for me 20 years ago.
And I ran 1% of web traffic with an ops team of 5 back then. So, idk what's going on here.
EDIT: Likely people are flagging me because I sound arrogant (or I hurt their feelings by talking bad about YAML-ops), but all I am doing is answering the question presented based on my experience.
Azure repos are kinda fine. It's really basic and there is nothing to break. I actually really really like their ticketing thingy for the same reason. It has the necessary stuff and the management types can't add a million of fields to it and annoy me with reporting, burndown charts or what not.
I'm on the other side of the fence. We're just about done migrating from GitHub to GitLab (self-hosted) and it's been refreshing to DGAF about any of the GH outages I read about.
Similar boat myself too, finished moving all important stuff from GitHub to self-hosted Forgejo with cross-platform builds. Not only do I avoid all the downtime stuff, but E2E builds also takes ~20% of the completion time it used to take, since now my runners have dedicated hardware hosted at home.
To maintain a fair comparison, GitHub has supported self-hosted runners for several years (maybe that doesn’t work for your specific usage, for whatever reason).
> To maintain a fair comparison, GitHub has supported self-hosted runners for several years
Yeah, tried that first, as I didn't want to move to Forgejo, I just wanted to keep working when I wanted to work.
The GitHub runner on Linux seemed fine, but the ones for macOS and Windows seemingly did something that made them a hell lot slower than even running VMs and then executing stuff inside those. I'm not sure what the runner is doing, if there is some built-in sandboxing or what not for those platforms, but it wasn't feasible to rely on for me as the builds took way too long time.
We were on self-hosted Gitlab but after a merger were forced to Github. Navigation feels painful in comparison and basic features such as commit graph are now behind more expensive tiers.
Interesting! I worked with Gitlab and I also thought it was quite clunky. If it was not for the stability issues GitHub is fine. Any other alternatives to GH or GL?
> Honest question, why are companies interested in hosting on github?
Mostly boils down to marketing and easier to establish a community. Almost every developer has an account there, leading to network effects being much larger, so if you're a new FOSS project, finding contributors and getting your project in front of other's eyes is much easier when you're on GitHub compared to your own Forgejo instance.
With that said, I'd question if chasing "most external one-time contributors" or GitHub stars is the right way to actually run a FOSS project, personally I'd avoid thinking about those vanity-numbers as much as possible and focus on the project, code and contributors themselves.
But, I've literally heard those two arguments for "why GitHub" countless of times over the years.
Usually, at large enough corporations, it's one of two things. Some random project gets open sourced, and it ends up on Github(see, for example, Salesforce) - or, more commonly, some subsidiary or acquisition had github and has either refused to migrate to the internal source system or the hassle of migration isn't worth it.
Go with the flow, don't rock the boat and use what developers already know, are probably the most cited reasons I've heard.
I've tried so many times in the past to argue for self-hosted setup that you fully control if you can afford it, things just get so much smoother and if you're a software development company, you probably want to own the software development workflow E2E so you can actually ship as fast as you want.
Why do you care about github? It’s Just another corporation doing what they know best: harvesting money. The software ecosystem can live without github just fine
With Ghostty being the latest project to leave GitHub, it does make me wonder who will leave next.
I don't expect everybody and their nan to leave GitHub by next wednesday and spin up their own Forgejo server, but I do think GitHub should be worried that people are finally looking to move away from them.
Titles can be genrated as well, just tell the LLM whether your readers love drama and witch hunts or are pseudo intellectuals. It'll come up with the correct framing.
M$ bought GH almost a decad ago. Instability is more due to mainstream traffic, combined (+ AI automation pushes on pointless repos) than some slow motion evil mastermind plot. Hanlon's razor applies.
I feel like I’m out of the loop, or maybe I’m just not a super GitHub power user, but GitHub does pretty much what I expect and I haven’t had issues with it. All my git commands for GitHub just work and PRs and code reviews are the same as it’s always been.
Can someone explain what exactly is so bad now that leaving it entirely to use some new platform, even spinning up your own servers, is a reasonable alternative?
It's more if you use it for things beyond traditional dev work. GitHub Actions have become very unstable plus someone using it at this level where people are trying to download/ file issues/ send code up 24/7 would feel the pain of every outage, not just those that happen during one's working hours.
I’m a relatively casual user of GitHub and even I’ve run into availability issues when pushing up changes. Your comment makes it sound like you don’t use GitHub much at all or maybe are in some time zone or AZ that’s somehow insulated.
When using it every day (and especially when using Github Actions), there's something broken or half-broken nearly every day.
Most random errors in Github Actions (e.g. jobs just randomly failing or getting stuck and requiring a manual restart, or just being plain slow) also never show up on the Github Status page. The Github Actions VMs are also so slow that I'm seriously pondering setting up a cheap throw-away laptop at home as runner, that would easily be 10x faster. But then we're at playing IT admin at home :/
> and PRs and code reviews are the same as it’s always been
When they changed the PR view to not display all the changes at once, was the moment I said "I really need to find something else", not only is the platform very unreliable (at least from Spain), but most product changes they do are making the platform less efficient for me as a developer to use.
> Can someone explain what exactly is so bad now that leaving it entirely to use some new platform, even spinning up your own servers, is a reasonable alternative?
It always was, but network-effect of GitHub been large. But seemingly not infinite, at one point people start favoring "Being able to access platform" over "people can star my repository" it seems.
something reliably breaks 7 am PST (sometimes earlier) if you're using anything more than the git command line and sometimes (not too often, true) even the git protocol breaks.
They have less uptime than my home NAS, this is the #1 thing that is wrong with it.
And the most recent bug is "we added random code to some PR and removed random code from others" which is a wtf bug which should be impossible to exist.
It's probably due to a more recent change, IE, focusing on features over stability. Or, it could be that there was some turnover in ops and someone who was a hawk about stability isn't there.
If I were to bet, there's probably a product manager or other leader who's just gung-ho on new features and loosing track of who their customers are and what their needs are.
That can happen many times during a buyout. Some company buys a thing. The problem then is ownership of the thing. Who in the new company is going to own the 'make sure it stays good' problem. Sometimes with a buy out the people who were doing that may even stay at the company. But it is a matter of motivation. MS has a real serious problem. You can see the gaps where they have glued together at least 10 companies together and called it microsoft. They have a huge reputational risk issue. Where something breaking in the xbox div can have a negative impact on the tools division. Also the other way around. They lack focus on many items. They have needed a 'service pack 2' stop the presses moment and fix this mount everest of tech debt.
I'm not sure it's specific to vibe coding so much as the AI feature add rush. Every SAAS company is throwing more shit at the wall than I've ever seen, to the point where I'm actively avoiding some software because I don't want yet another new feature release pop-up when I log in.
Add in them being extremely high scale and critical infrastructure and it's easy to see where things can go wrong, vibe added code or not. I think we'd all prefer they have long slow roadmaps but clearly leadership thinks they're in a fight with the other AI companies to release the newest and bestest every day.
Definitely not, I remember some 4 years ago some random bug in a github-supported github-action and a comment in an issue saying: "I heard the team responsible for this action was laid off, don't expect a fix". This was shortly after the microsoft acquisition.
But the vibe coding BS probably made it 10 times worse.
They started with a hands off approach and then went hands on, I’m not sure but that ‘hands on’ timing is likely to happen shortly after the usual acquisition vesting period of 3 years when the old guard starts to leave.
> But the vibe coding BS probably made it 10 times worse.
Yup, keep seeing this in various companies. Teams that were effective and did solid engineering now are more effective and does even better engineering. Teams that were effectively already just "boilerplate monkies" now produce a lot more code than before, but the quality is the same so effectively they're worse at contributing now than before, and take more shortcuts, not less.
From my point of view, agents are amplifiers, so if you usually build spaghetti projects, agents just help you do that faster, not avoid the spaghetti altogether. If you usually build well-designed stuff, they can help you put that together faster.
Agreed. In general the amount and variety of bugs introduced since everyone started vibing is worrying. It is probably a national security concern but I guess so is the economy tanking due to failed AI investments. Guess we will see
Perhaps they can’t help themselves out of habit, it is their nature.
The original red dog team that started azure is long gone and the general success of the cloud papers over all levels of incompetence so that the incompetence is now entrenched and unable to do better.
Cloud service providers have this unfortunate property where poor designs will make more money which makes it hard to maintain a culture of excellence. I tried to push a design change that would result in a 10x throughput for a certain product and was told that a 90% drop in usage is the last thing they want. I self host my own stuff with GitLab, so far not a single unplanned outage in 6 years.
I wonder if there's a place for something like matrix, but for repositories (or maybe matrix protocol can handle that?). A world where we have selfhosted, saas, etc, but all interlinked and searchable? Say I find a project on gitlab, I want to contribute to, I checkout to my personal server (or someone elses hosted), and raise a pr back to original repo.
I know it doesn't answer your question, just thinking aloud really.
This would be best, though it's a big ask, when everyone has gone from self hosting plus a sprinkling of cloud services, to only cloud services and no remembrance of how to self host.
I used to run a git server for all my main projects, and mirrored public ones on GitHub. Then the convenience of GitHub lured me in, to the point I shut down my private git server 5 years ago.
Let's be honest there's an order of magnitude or more higher throughput volume of PR jitter and new repo bloat which makes this look like a viral digital native at scale.. couple that with being owned by one of the most scale immature companies on the planet ... of course it's a problem.
Get these folks off Azure and Cosmos DB (or whatever MSFT forces them to use) to something real and maybe you'd have a shot
It isn't surprising at all, Microsoft is doing a PE firm playbook with what they buy. You don't need to look much far, let's think about its biggest acquisition to date, Blizzard.
Blizzcon canceled. All of its IP barely got any love.
See what players think about the latest World of Warcraft patch. It's absolutely shit and broken. People say they fired the entire QA department since a few years back and since then the quality has just gone down.
They buy those businesses because they have nothing to do with that free cash flow, and for accounting reasons it makes sense to have them.
They didn't buy those businesses to develop it further and make it worth more.
Github will just become ever more irrelevant.
The key issue is that the US governments let those huge monopolies exist, and then use their money to buy other businesses and enshiftify them.
Unless that changes in the US, this will continue happening.
> See what players think about the latest World of Warcraft patch. It's absolutely shit and broken
Crazy to me that the loot tables are still broken for some players/characters, they've tried to fix it several times now, and it's still not working - Since (some) endgame gear can only be obtained this way they've effectively soft locked those players/character out of the endgame.
Context: Some players are always receiving the same drops i.e. a belt. Rather than a varied loot table that gives them a chance to get items they need.
The wildest thing is that Azure Repos/Pipelines was better than this.
Their one caveat is also that they are still migrating it to Azure infra, so it's possible that's still in a one foot in one foot out kinda scenario, from what I've heard. But, this isn't inspiring confidence.
They mentioned they have some elasticsearch reindexing going to, I would guess they needed to regard or move stuff and something didn't work well. But if I understood it right they mentioned the PRs ES index which they didn't shared proof increased as the number of repos.
It might be anything. It seems they lost huge chunks due to layoffs and structural changes and MS which has the reverse golden Midas touch.
This is just pure speculation but also now there is no reason for MS to keep GH working. They absorbed all code they wanted. Now they can let it burn. Would be even better for them if that happened
But the 1% of repos that do have PRs and actions are likely going to be seeing enormous increases in volumes
I have been a part of two very large companies with self hosted gits and I've seen enough to be confident that this is an incredibly hard thing to manage
Anyone who has been part of that journey knows how painful it really is. A lot of times the systems to fail at all levels, and you have to redesign it from the first principles.
[1] https://github.blog/news-insights/company-news/an-update-on-...
And that start by layoffing your best engineers, I guess
I have, but it depends what you mean.
Scenario 1: e-commerce SaaS (think: Amazon but whitelabel, and before CPUs even had AES instructions); Christmas was "fun".
Scenario 2: Video Games. The first day is the worst day when it comes to scale. Everything has to be flawless from day 0 and you get no warning as to what can go wrong.
Yet, somehow, I managed to make highly reliable systems.
In scenario 1; I had an existing system that had to scale up and down with load, this was before there was cloud and hardware had a 3-4 month lead time, so most of the effort was around optimising existing code, increasing job timeouts and "quenching" sources that were expensive. We used to also do so 'magic' when it came to serving requests that had session token or shopping cart cookie.
In scenario 2; we have a clean-room implementation and no legacy, which is a blessing but also a curse, there's no possibility to sample real usage: but you also don't need to worry about making changes that are for the better.
So, pro's and con's... but it's not like handling huge load hasn't been done before, computers are faster than they ever have been and while my personal opinion is that operational knowledge is dying (due to general distain for people who actually used to run systems that scale: not just write hopeful "eventually consistent" yaml that they call deterministic) - the systems that do exist today hold your hand much better than they did for me 20 years ago.
And I ran 1% of web traffic with an ops team of 5 back then. So, idk what's going on here.
EDIT: Likely people are flagging me because I sound arrogant (or I hurt their feelings by talking bad about YAML-ops), but all I am doing is answering the question presented based on my experience.
[1] https://bloomberry.com/data/github/
What made it better than e.g GitLab?
https://news.ycombinator.com/item?id=47616242 https://isolveproblems.substack.com/p/how-microsoft-vaporize...
https://www.youtube.com/watch?v=Js_Y_q-IkYo
Yeah, tried that first, as I didn't want to move to Forgejo, I just wanted to keep working when I wanted to work.
The GitHub runner on Linux seemed fine, but the ones for macOS and Windows seemingly did something that made them a hell lot slower than even running VMs and then executing stuff inside those. I'm not sure what the runner is doing, if there is some built-in sandboxing or what not for those platforms, but it wasn't feasible to rely on for me as the builds took way too long time.
- SourceHut: https://sr.ht/~sircmpwn/sourcehut/
- Forgejo (used by Codeberg, etc.): https://forgejo.org/
At a customer we're implementing GitHub Actions and even on our Dev environment there are so many hickups with GitHub.
Might be pricy though.
MSFT should just create slophub.com they'd make money im sure.
As a private person I use it too as a free hoster, but from work I mainly know self hosted instances of jenkins and TeamCity.
I guess, but it's not like you can't learn how to create a pullrequest on bitbucket or how to create an issue on jira as well within a work day?
That seems like the smallest thing when switching to a new company.
> The friction for adopting features like Actions is relatively low.
Yeah, I know almost nothing about the CI integration and actions when it comes to Github. Will look into it. Thank you.
Mostly boils down to marketing and easier to establish a community. Almost every developer has an account there, leading to network effects being much larger, so if you're a new FOSS project, finding contributors and getting your project in front of other's eyes is much easier when you're on GitHub compared to your own Forgejo instance.
With that said, I'd question if chasing "most external one-time contributors" or GitHub stars is the right way to actually run a FOSS project, personally I'd avoid thinking about those vanity-numbers as much as possible and focus on the project, code and contributors themselves.
But, I've literally heard those two arguments for "why GitHub" countless of times over the years.
But closed source companies surly don't need to establish a community?
I've tried so many times in the past to argue for self-hosted setup that you fully control if you can afford it, things just get so much smoother and if you're a software development company, you probably want to own the software development workflow E2E so you can actually ship as fast as you want.
I don't expect everybody and their nan to leave GitHub by next wednesday and spin up their own Forgejo server, but I do think GitHub should be worried that people are finally looking to move away from them.
https://github.com/SerJaimeLannister/who-left-gh/
Currently I know 3 projects, Ghostty, Bookstack-app, Hardenedbsd who have seemed to move away from Github from my understanding.
https://ziglang.org/news/migrating-from-github-to-codeberg/
Such a one punch sentence that distills the message with a little bit of dramatic flair.
got damn, anyone got recommendations on how to write like a journalist ?
The rest of the article can be AI generated, don't fret about it.
the fake surprise is so fake
Can someone explain what exactly is so bad now that leaving it entirely to use some new platform, even spinning up your own servers, is a reasonable alternative?
So much so that they stopped posting uptime metrics for a while on their status page and an independent 3rd party created a website just for this:
https://mrshu.github.io/github-statuses/ (not my website)
According to that website, which unsurprisingly reports a lower number than what Github themselves claim, Github uptime is down to ~86%.
And if you work in the space, you know how terrible that is, but even more so for such a critical piece of infrastructure.
Most random errors in Github Actions (e.g. jobs just randomly failing or getting stuck and requiring a manual restart, or just being plain slow) also never show up on the Github Status page. The Github Actions VMs are also so slow that I'm seriously pondering setting up a cheap throw-away laptop at home as runner, that would easily be 10x faster. But then we're at playing IT admin at home :/
When they changed the PR view to not display all the changes at once, was the moment I said "I really need to find something else", not only is the platform very unreliable (at least from Spain), but most product changes they do are making the platform less efficient for me as a developer to use.
> Can someone explain what exactly is so bad now that leaving it entirely to use some new platform, even spinning up your own servers, is a reasonable alternative?
It always was, but network-effect of GitHub been large. But seemingly not infinite, at one point people start favoring "Being able to access platform" over "people can star my repository" it seems.
And the most recent bug is "we added random code to some PR and removed random code from others" which is a wtf bug which should be impossible to exist.
If I were to bet, there's probably a product manager or other leader who's just gung-ho on new features and loosing track of who their customers are and what their needs are.
Add in them being extremely high scale and critical infrastructure and it's easy to see where things can go wrong, vibe added code or not. I think we'd all prefer they have long slow roadmaps but clearly leadership thinks they're in a fight with the other AI companies to release the newest and bestest every day.
But the vibe coding BS probably made it 10 times worse.
The acquisition was 8 years ago.
Yup, keep seeing this in various companies. Teams that were effective and did solid engineering now are more effective and does even better engineering. Teams that were effectively already just "boilerplate monkies" now produce a lot more code than before, but the quality is the same so effectively they're worse at contributing now than before, and take more shortcuts, not less.
From my point of view, agents are amplifiers, so if you usually build spaghetti projects, agents just help you do that faster, not avoid the spaghetti altogether. If you usually build well-designed stuff, they can help you put that together faster.
Embrace, extend, and extinguish.
What exactly are they extinguishing GitHub to the benefit of? Azure Repos?
The original red dog team that started azure is long gone and the general success of the cloud papers over all levels of incompetence so that the incompetence is now entrenched and unable to do better.
Cloud service providers have this unfortunate property where poor designs will make more money which makes it hard to maintain a culture of excellence. I tried to push a design change that would result in a 10x throughput for a certain product and was told that a 90% drop in usage is the last thing they want. I self host my own stuff with GitLab, so far not a single unplanned outage in 6 years.
I used to run a git server for all my main projects, and mirrored public ones on GitHub. Then the convenience of GitHub lured me in, to the point I shut down my private git server 5 years ago.
Now I kinda regret that decision.
Get these folks off Azure and Cosmos DB (or whatever MSFT forces them to use) to something real and maybe you'd have a shot
Blizzcon canceled. All of its IP barely got any love.
See what players think about the latest World of Warcraft patch. It's absolutely shit and broken. People say they fired the entire QA department since a few years back and since then the quality has just gone down.
They buy those businesses because they have nothing to do with that free cash flow, and for accounting reasons it makes sense to have them.
They didn't buy those businesses to develop it further and make it worth more.
Github will just become ever more irrelevant.
The key issue is that the US governments let those huge monopolies exist, and then use their money to buy other businesses and enshiftify them.
Unless that changes in the US, this will continue happening.
https://blizzcon.com/en-us/event/
Crazy to me that the loot tables are still broken for some players/characters, they've tried to fix it several times now, and it's still not working - Since (some) endgame gear can only be obtained this way they've effectively soft locked those players/character out of the endgame.
Context: Some players are always receiving the same drops i.e. a belt. Rather than a varied loot table that gives them a chance to get items they need.
I think Diablo Immortal was likely the biggest success Blizz provided there
He's blaming GitHub that he can't get work done?