Tell HN: HN was down

- HN errored on all authenticated requests with 502 Bad Gateway. It did still respond to a limited amount of unauthenticated requests with presumably cached pages, which did not get updated. The last post on /newest claimed "0 minutes ago", but was actually much older (1:32:57 PM GMT) and not the newest post.

- This status page actually identified the outage: https://hackernews.onlineornot.com/ - Pages by Hund and Statuspal did not show the outage.

- The last post before the outage was https://news.ycombinator.com/item?id=46301823 (1:39:59 PM GMT). The last comment was https://news.ycombinator.com/item?id=46301848 (1:41:54 PM GMT).

- There was an average of ~4 seconds per comment just prior to the outage. Based on this, HN likely went down at 1:41:58 PM GMT.

236 points | by uyzstvqs 1 hour ago

62 comments

  • dang 32 minutes ago
    Yes, sorry! We're investigating, but my current theory is we got overloaded because I relaxed some of our anti-crawler protections a few days ago.

    (The reason I did that is that the anti-crawler protections also unfortunately hit some legit users, and we don't want to block legit users. However, it seems that I turned the knobs down too far.)

    In this case, though, we had a secondary failure: PagerDuty woke me up at 5:24am, I checked HN and it seemed fine, so I told PagerDuty the problem was resolved. But the problem wasn't resolved - I was just sleeping through it.

    I'll add more as we find out more, but it probably won't be till later this afternoon PST.

    • shlomo_z 4 minutes ago
      Crazy that Dang literally manages HN in his sleep!

      We all knew that but I haven't seen any confirmation before this.

    • Rooster61 9 minutes ago
      No apology needed. We all needed to stop procrastinating anyways :)
    • echelon 15 minutes ago
      I didn't realize you were carrying the pager too! Kudos!
    • shmeeed 30 minutes ago
      Looking forward to the post mortem. :)
  • Elfener 1 hour ago
    I got stuck in an infinite loop.

    Try opening HN -> it's down, better check HN to see everyone talking about a major website being down -> Try opening HN -> loop

    • mustak_im 1 minute ago
      I woke up and was wondering if I’ve just woken up in hell!
    • HPsquared 32 minutes ago
      Sometimes I'll catch myself absentmindedly reopening the browser and checking two or three front pages, seconds after having just checked them and closed the browser.
    • neom 1 hour ago
      Yeah me too. Wake up -> HN down -> That's weird, oh well it's usually only down for a few minutes -> I should check if HN is still down -> That's weird, oh well it's usually only down for a few minutes -> I should check if HN is still down -> loop.

      That was a few hours ago. I'm glad this loop is broken.

      • squeefers 45 minutes ago
        sounds very much like an evil social media dopamine feedback loop. ironic given everyone on HN is so anti social media.... its clearly only bad for kids though i should add, silly of me to exclude such a detail
        • bee_rider 11 minutes ago
          HN is obviously social media and it is silly to say otherwise. It is just social media that occasionally has interesting stuff. The SNR is just slightly higher.
        • neom 37 minutes ago
          I've been on this internet hit shit since the 90s lil bro, s' all good.
          • strbean 35 minutes ago
            I can stop any time I want, I just don't want!
  • manbitesdog 1 hour ago
    TIL I have a "open Hacker News" hand reflex
    • directmusic 10 minutes ago
      I'm glad I'm not the only one. If I type 'n' into any browser it autocompletes to HN.
    • ectospheno 1 hour ago
      I learn more reading the comments here than anywhere else. Thanks everyone for my addiction.
    • 1shooner 1 hour ago
      If you're looking to put the brakes on that, I've used LeechBlock to add a 5-second timer to opening a new HN window (along with other block schedules). The timer even fails if it loses focus, so it really helps slow you down.

      https://www.proginosko.com/leechblock/

      • squeefers 44 minutes ago
        so youve got the willpower to do something about it but not enough to just stop doing it?
        • 1shooner 25 minutes ago
          Yes.
        • dwedge 39 minutes ago
          Have you never suffered from habitual reflexes? I blocked twitter for a while in my hosts file and a dozen times over those first few days I instinctively opened a new tab and typed twitter in
          • squeefers 10 minutes ago
            > I blocked twitter for a while in my hosts file and a dozen times over those first few days I instinctively opened a new tab and typed twitter in?

            youd go through that effort when you could have just stopped though.

    • jstummbillig 1 hour ago
      I say. Vibe coded 4 apps once I got past that, on my way to half a billion in ARR already.
    • thesurlydev 1 hour ago
      Same! Right there with "every day must begin with coffee"
    • lysace 35 minutes ago
      ⌘-T, N, <RET>

      Did it like 5 times during that 1h-ish outage. :(

    • wincy 1 hour ago
      I’ve turned on no procrast mode and set it to ten minutes per hour. Helped me a lot!
      • HanClinto 58 minutes ago
        What are you using to control this?
        • randallsquared 11 minutes ago
          It's a setting available on the page you get from clicking on your own username.
        • bee_rider 9 minutes ago
          It is on your profile, the “noprocrast” dropdown.
    • AndrewKemendo 1 hour ago
      It just reinforces for me that addiction is a human problem not a problem with technology

      I know dang basically works tirelessly to not change the format in order to not induce those addictive patterns

      but yet here we all are

      • chistev 1 hour ago
        It's a website with the smartest people in the world. The level of conversations here are unrivaled in internet communities.

        It's understandable to be addicted. Lol.

        I visit this place multiple times a day.

        • steve_adams_86 31 minutes ago
          > the smartest people in the world

          But also, people like me. Be careful what you choose to believe on this website

        • phantasmish 1 hour ago
          99% of social science or political topics and 50% of technical topics here do not… read as smart, and you’d be much better off spending the same time reading the first chapter of a relevant 101-level college textbook.
          • seizethecheese 50 minutes ago
            It's entirely possible that this is the smartest place on the internet, but also often dumb. In fact, it seems likely. More of an indictment of the rest of the places on the internet.
            • squeefers 40 minutes ago
              > It's entirely possible that this is the smartest place on the internet,

              i cant find the link, but there was a post about how to "be nice" and it was a revelation to a worrying amount of "geniuses" on here. bare in mind the sum total of the advice was "be nice, dont be rude"

        • fwip 1 hour ago
          It's really not 'the smartest people.' It's people interested in tech, and often in making-a-lot-of-money-in-tech. It does have a lot of people with significant industry experience, which is cool.
          • worksonmine 1 hour ago
            > It's really not 'the smartest people.'

            This was especially obvious during Covid, I even stopped visiting because the comment section was so crazy.

        • linhns 39 minutes ago
          Also the level of flak is unrivaled.
        • andrepd 2 minutes ago
          Poe's Law for a parody of the self-important sv techbro
        • the_af 1 hour ago
          > It's a website with the smartest people in the world.

          Nice joke!

          At least, I hope it was a joke...

      • PurpleRamen 1 hour ago
        But, would the addiction become worse if HN changed, or would there be a point where they could cure it?
      • dzink 1 hour ago
        This one is at least healthy-ish for the mind. I’d much rather hacker news than any other news. Social Media is an emotional rage-bait cesspool these days. If it’s not for Hacker News those of us who abstain from the rest would be living in the dark.
    • ChrisMarshallNY 1 hour ago
      So do I, but it was such a shock that I just passed out, and when I woke up, it was back up.
    • selectnull 58 minutes ago
      I already knew that. :)
    • kevin061 1 hour ago
      I did not know how addicted I was to HN until today lol
    • numpad0 52 minutes ago
  • sosodev 1 hour ago
    Yes, and I'm a little ashamed to admit my morning routine wasn't the same without it.
    • fedreg 1 hour ago
      This was more impactful to my day than the last AWS and CloudFlare outages...
      • messe 1 hour ago
        At least during those outages I could procrastinate on HN.
        • the_arun 40 minutes ago
          I felt like changing HN down page to show top 30 posts from this week before or after the generic message.
  • al_borland 1 hour ago
    Is this still a valid account for HN status? It says it’s the official one, but with the changes at Twitter to no longer show chronological feeds (at least for users that aren’t logged in), it’s rather useless. The top 5 listed post (for me) are seemingly random from 2014 - 2022.

    https://x.com/HNStatus

    Is there a better place to check, beyond a basic down detector that may provide more insight or signal that the outage is acknowledged?

    • FuriouslyAdrift 1 hour ago
      Only way I have figured out how to to change the "Following" sort order back to chronological is from the mobile app: click the down arrow on the "Following" tab. Change the sort from "popular" to "most recent."

      Seems to reset it on the web view, too.

      • al_borland 14 minutes ago
        It sounds like this would only work for logged in users.
    • zipy124 1 hour ago
      https://hn.hund.io/ Is a status page, no idea if official or not, but it didn't register here for some reason.

      I didn't read the post text, it's identified there haha, my bad! I wish the text post text wasn't grey, I gloss over it too easily.

      • laCour 50 minutes ago
        This was monitoring the unauthenticated news page, which is why it didn't catch it. It now monitors authentication as well. It is not official, and was made by a co-founder years ago.
      • lagniappe 1 hour ago
        This site said HN was fine and green the entire time it was down.
  • laCour 52 minutes ago
    I'm with Hund. Our hn.hund.io page did not catch this because it was requesting the cached, unauthenticated page. It now monitors authentication as well.
    • joncrane 50 minutes ago
      Thank you. I was thinking myself or my corporate IP was shadowbanned
    • jonahx 51 minutes ago
      Is this a mistake by hund, or the configuration of hund by HN?
      • laCour 48 minutes ago
        Mistake on our part (Hund) for not monitoring authentication. This page is unofficial and was made by a co-founder several years ago.
  • liampulles 1 hour ago
    Smart. Have to use that error budget before year end...
    • dylan604 1 hour ago
      I always hated the late use-it-or-loose-it at the end of the year where you end up buying the things that were denied requests from earlier in the year. You just cost me half a year of using the damn thing.
  • steve_adams_86 27 minutes ago
    I was fortunate enough to get to watch a bunch of kids racing and bouncing in bouncy castles in our school gym during HN's downtime.

    It's not that much different from HN, come to think of it.

    (ha, ha)

  • xxs 37 minutes ago
    PSA - if you delete your cookies, HN gets it easier. Or just test it in a private window.

    It did work without being logged on. The auth service appeared to be down as the log in attempt (just showing the page) failed.

    • scottydelta 28 minutes ago
      Now it makes sense. I was puzzled about why it was working on the phone browser and not on my system. I'm logged into HN on my system.
  • wavemode 1 hour ago
    When it was down my thought was "damnit, I'll actually have a productive workday now."
    • tzs 1 hour ago
      Next time you can avoid that fate by opening HN in a private browsing (or whatever your browser calls its equivalent) window. This outage, like the vast majority of HN outages, only affected logged in requests.

      I suppose you could also just clear your HN cookies in regular browsing window, but then when they fix it you'd have to log in again.

      • dpoloncsak 33 minutes ago
        Huh. Dunno why, but when it failed on Firefox I tried Chrome, and it worked. I wrote it off as a Mozilla issue, but this would better explain that I think
      • neom 1 hour ago
        Is it my imagination or did they used to automatically serve you a logged out page when it was down?
    • dwa3592 1 hour ago
      hahaha
  • esafak 1 hour ago
    A lot of the outage indicators failed. Someone needs to create an outage indicator reliability dashboard.
    • Nextgrid 1 hour ago
      A lot of them got fooled by the caching; pages for signed-out users are cached heavily and those kept returning successful responses even if the actual backend server was down.
    • chistev 1 hour ago
      And an outage reliability indicator for that outage reliability indicator.
      • willis936 54 minutes ago
        We apologize again for the fault in the fault indicator. Those responsible for sacking the people who have just been sacked have been sacked.
    • arionmiles 1 hour ago
      So a downdetectorsdowndetector.com but for Hacker News?
  • ortusdux 1 hour ago
    • rozenmd 1 hour ago
      Interestingly it stayed up if you weren't logged in.
      • jedberg 1 hour ago
        If you aren't logged in you get a cached version from the CDN/cache. Reddit works the same way.
      • Izkata 1 hour ago
        Not completely, I'm not logged in on my work laptop and it was only working some of the time (and not like some pages were cached and some weren't, I was refreshing the same page and sometimes it worked and sometimes not).
        • cess11 1 hour ago
          That's how I concluded that it wasn't a ban on my account but rather more serious.
      • bryanrasmussen 1 hour ago
        also went down if you went to login, and people's individual pages were also down. So as far as I saw the front page was up as long as you were not logged in, however I'm not sure if that wasn't just luck of the draw, I had one experience where it looked like maybe the front page was sometimes down for not logged in users as well.

        on edit: ok others pointed out it was cached pages I saw. explains it.

      • smallerize 1 hour ago
        That only worked for a while, eventually I couldn't load comment pages even logged out.
      • davnicwil 1 hour ago
        that'll be because it's served from cache when you're not logged in
  • danielfalbo 1 hour ago
    As soon as I noticed it was down I came to hacker news to post about it, but...
  • sltr 1 hour ago
    Yeah I couldn't log in for a bit this morning. It's concerning how often and how many times I tried. Glad it's resolved.
    • imiric 1 hour ago
      Next time try not to beat a dead horse expecting it to resurrect. :)
      • mlhpdx 1 hour ago
        So now we call the refresh button CPR?
  • dhruv3006 1 hour ago
    I was quiet surprised really. HN almost never goes down.
    • Aachen 1 hour ago
      You don't visit enough my friend :)

      It's down about 8.4 minutes per week. On 26% of days it doesn't work at least once, and on 12% of days it has more than one consecutive failed check. The longest uptime streak was 24 days

      I've been keeping track since exactly 2 years (to the day!) because I was surprised that it seemed briefly down for me on a daily basis. Was I getting unlucky and hitting it every time, or was it just down very often? Nobody posted anything so I started answering the question for myself :p

      I've been meaning to post the tracker to HN but there's a pesky bug I want to fix: the "is it currently down" stat. I don't know how this is beyond me but something in the code bugs out. So this is my first time posting about it

  • ashf023 1 hour ago
    Yes, and on one request I saw a message like "Restarting server - this won't take long", and soon after it's back up.
  • zzixp 1 hour ago
    First AWS/Azure/Cloudflare and now HN?!?
  • rob 1 hour ago
    Maybe PG is more involved now and hired the "10,000 lines of AI code a day" person who made a deployment mistake?

    https://x.com/paulg/status/1953289830982664236?s=46

  • ajdude 1 hour ago
    I was able to view the site without being signed in (i.e. private window) but any browser I was logged into wouldn't load.

    I'm sure it's a coincidence but it started working again shortly after emailing hn@ycombinator.com

  • oidar 46 minutes ago
    What was the longest that HN has been down? I feel like this is up there.
  • arm32 47 minutes ago
    The "site status" apps are all smoke and mirrors, how unreliable.
  • chistev 1 hour ago
    It was the first time since I started using this website (August last year) that it was down.

    I'm still impressed nonetheless.

    I'd like to know what caused the outage and how it could have been prevented, for learning purposes.

  • Kim_Bruning 1 hour ago
    In with the rest, yes, and my first thought is always "I have an internet outage" when HN is down. :-P
  • stevenjgarner 1 hour ago
  • PurpleRamen 1 hour ago
    In other words, productivity in tech skyrocketed for hours..though it seems some work was flavoured with irrational anger.
  • bossyTeacher 13 minutes ago
    I have never before seen this website down.
  • dvaun 55 minutes ago
    Will we get a post mortem?
  • fatty_patty89 1 hour ago
    I got irrationally angry when it refused to load the website
    • imvetri 1 hour ago
      you could refuse to be angry to mock it
    • DonHopkins 1 hour ago
      You handle your irrational anger much better than whatevermrfukz who keeps pooping his pants.
      • fatty_patty89 1 hour ago
        Interesting that you mention that, how do you keep track of his comments? Some sort of plugin/scraper?
        • Jtsummers 1 hour ago
          > how do you keep track of his comments?

          You can just look at them, turn on showdead in your profile and you'll see a bunch of flag-killed comments in this discussion by whatevermrfukz. No need for a plugin or scraper.

  • arbirk 1 hour ago
    We need a serious post mortem for this
  • zelphirkalt 1 hour ago
    Yep "having trouble serving my request" or so.
  • numpad0 1 hour ago
    Could have been accidental flagging of sorts. Didn't work on PC for few minutes while showing fine on phone.
  • sangeeth96 1 hour ago
    Thought it had something to do with some model updates, like Gemini Flash 3 for a moment.
  • pedro380085 1 hour ago
    Yes, I got an error message, but I cleared my cookies and was able to access.
  • neurolesudiste 1 hour ago
    Hi, Got an error an hour ago on phone, not loged.

    Anyway, glad to see you back.

    Paris 1812.

    Cheers from France.

  • voxleone 1 hour ago
    Yes, it' been out for me too, southern hemisphere, GMT -03
  • rustaceanU32 1 hour ago
    I'll admit this ruined my morning
  • gepiti 1 hour ago
    Athens, Greece yes it was down.
  • neurolesudiste 1 hour ago
    Yup, France here , Paris 1809.

    HN was down about an hour ago.

    Glad to see it back !

    Cheers.

  • silverpiranha 1 hour ago
    Yes, was down for me too this morning
  • gaigalas 1 hour ago
    I got confused by the "minutes ago" thing.

    Working with full dates in the HTML and doing a tiny JavaScript that calculates the "minutes ago" would actually be a neat improvement.

  • markus_zhang 1 hour ago
    Yes was down for me. Ontario, Canada.
  • wek 1 hour ago
    Yes, it was down for a few hours
  • thatgerhard 1 hour ago
    Thought for a second I got banned for something lol
    • elxr 1 hour ago
      I thought I was being rate-limited for opening posts too fast, which has happened before.

      After more than an hour I thought, "wow this is pretty harsh" and "so much of my exposure to learning things is directly tied to HN posts". I was lost lol.

    • Aachen 54 minutes ago
      Your comment history doesn't sound IP-ban-worthy. Is there an alt that has you worried? :P
    • verzali 43 minutes ago
      Me too and I was wondering what I did!
  • agumonkey 1 hour ago
    hn applied ycombinator too strictly
  • SilverElfin 1 hour ago
    I tried to refresh an embarrassing amount of times
  • russellbeattie 1 hour ago
    HN being down makes you start wondering about the differences between routine, addiction, compulsion, and habit.
  • khaledh 1 hour ago
    It was down if you tried to access it while authenticated (i.e. you have a cookie). It was loading fine for unauthenticated sessions (e.g. incognito).
  • rzerowan 1 hour ago
    Yes
  • busymom0 1 hour ago
    I run the app called HACK and received user emails that the HN website was down.
    • maverwa 1 hour ago
      Thank you for HACK. I love it!
  • spooneybarger 1 hour ago
    Yes
  • ProofHouse 1 hour ago
    I wake up, and look at HackerNews lol. This morning sucked hahaha
  • bennydog224 1 hour ago
    Yes, I had to touch grass this morning.
  • FergusArgyll 1 hour ago
    Yeah & chatgpt was trying to gaslight me - claiming it was my fault. Happy to put that bum in its place...
    • chistev 1 hour ago
      You use ChatGPT to know it a site is down?
  • pcdoodle 1 hour ago
    [dead]
  • whatevermrfukz 1 hour ago
    [flagged]
  • whatevermrfukz 1 hour ago
    [flagged]
  • whatevermrfukz 1 hour ago
    [flagged]
  • whatevermrfukz 1 hour ago
    [flagged]
    • DonHopkins 1 hour ago
      I remember my first beer.
      • null_deref 1 hour ago
        That Markov chain spiraled out of control
        • rolandog 1 hour ago
          Perhaps Grok was not trained to go so much time with 0 responses due to the downtime?
  • gjsman-1000 1 hour ago
    Just a month ago, I got downvoted to -2 for saying HN for being self-hosted hasn't shown up as more reliable than something behind Cloudflare. My point is made.

    Edit: Now it happens again. Knee jerk defenses all the way down.

    • Aachen 48 minutes ago
      Keep in mind that votes aren't supposed to be about whether you agree or disagree, but whether a comment adds substance to the thread. A good question that makes a wrong assumption can be worth upvoting; a correct but fairly irrelevant statement about politics not (to give a relatively obvious example)

      Being "voted to -2" doesn't necessarily mean you were wrong (it often correlates though). People might just think it wasn't relevant in whatever context you posted it in

      I often find it hard to tell what makes people think something I write is not helpful (or sometimes also a comment someone else made) and thus appreciate comments that clarify constructively. It can also help to ask for clarification if you're particularly surprised about the votes on a given post

    • abujazar 1 hour ago
      Unless the downtime was caused by something Cloudflare would've prevented, this downtime would've happened regardless of being behind Cloudflare. Cloudflare adds another single point of failure.
    • null_deref 1 hour ago
      Could this be a self-inflicted bug? In that case, the broader point still stands: cloud providers can cause outages that are outside your direct realm of responsibility.
      • gjsman-1000 1 hour ago
        Your VPS server and your data center and the ISP your data center uses and the AS system your ISP uses all can cause outages outside your direct realm of responsibility.
    • Jeremy1026 29 minutes ago
      You're probably being downvoted today for your "I was smarter than you" tone.
    • Jtsummers 1 hour ago
      Complaining about downvotes usually gets you more downvotes, so the response here isn't really that surprising.
    • DonHopkins 1 hour ago
      It's absolutely irresistible downvoting people who preemptively complain about being downvoted like you do. It really made my day. Post another complaint so I can do it again please! It's not knee jerk when you explicitly ask for it, by leading with a complaint about downvoting, instead of just making your point and letting it fall or rise on its own merits. You're the one who put the idea of downvoting you into my head in the first place.