Big-Endian Testing with QEMU

(hanshq.net)

40 points | by jandeboevrie 2 hours ago

7 comments

  • AKSF_Ackermann 1 hour ago
    > When programming, it is still important to write code that runs correctly on systems with either byte order

    What you should do instead is write all your code so it is little-endian only, as the only relevant big-endian architecture is s390x, and if someone wants to run your code on s390x, they can afford a support contract.

    • addaon 34 minutes ago
      There's still at least one relevant big-endian-only ARM chip out there, the TI Hercules. While in the past five or ten years we've gone from having very few options for lockstep microcontrollers (with the Hercules being a very compelling option) to being spoiled for choice, the Hercules is still a good fit for some applications, and is a pretty solid chip.
    • GandalfHN 14 minutes ago
      Outsourcing endianness pain to your customers is an easy way to teach them about segfaults and silent data corruption. s390x is niche, endian bugs are not.

      Network protocols and file formats still need a defined byte order, and the first time your code talks to hardware or reads old data, little-endian assumptions leak all over the place. Ignoring portability buys you a pile of vendor-specific hacks later, because your team will meet those 'irrelevant' platforms in appliances, embedded boxes, or somebody else's DB import path long before a sales rep waves a support contract at you.

    • j16sdiz 57 minutes ago
      If you comes to low level network protocol (e.g. writing a TCP stack), the "network byte order" is always big-endian.
      • skrtskrt 44 minutes ago
        Prometheus index format is also a big-endian binary file - haven’t found any reference to why it was chosen.
    • nyrikki 45 minutes ago
      The linked to blog post in the OP explains this better IMHO [0]:

         If the data stream encodes values with byte order B, then the algorithm to decode the value on computer with byte order C should be about B, not about the relationship between B and C.
      
      One cannot just ignore the big/little data interchange problem MacOS[1], Java, TCP/IP, Jpeg etc...

      The point (for me) is not that your code runs on a s390, it is that you abstract your personal local implementation details from the data interchange formats. And unfortunately almost all of the processors are little, and many of the popular and unavoidable externalization are big...

      [0] https://commandcenter.blogspot.com/2012/04/byte-order-fallac... [1] https://github.com/apple/darwin-xnu/blob/main/EXTERNAL_HEADE...

    • bear8642 1 hour ago
      > the only relevant big-endian architecture is s390x

      The adjacent POWER architecture is also still relevant - but as you say, they too can afford a support contract.

    • EPWN3D 1 hour ago
      I mostly agree, but network byte ordering is still a thing.
  • bluGill 28 minutes ago
    What I really want is memory order emulation. X86 as strong memory order guarantees, ARM has much weaker guarantees. Which means the multi-threaded queue I'm working on works all the time on development x86 machine even if I forget to put in the correct memory-order schematics, but it might or might not work on ARM (which is what my of my users have). (I am in the habit of running all my stress tests 1000 times before I'm willing to send them out, but that doesn't mean the code is correct, it means it works on x86 and passed my review which might miss something)
  • electroly 1 hour ago
    > When programming, it is still important to write code that runs correctly on systems with either byte order

    I contend it's almost never important and almost nobody writing user software should bother with this. Certainly, people who didn't already know they needed big-endian should not start caring now because they read an article online. There are countless rare machines that your code doesn't run on--what's so special about big endian? The world is little endian now. Big endian chips aren't coming back. You are spending your own time on an effort that will never pay off. If big endian is really needed, IBM will pay you to write the s390x port and they will provide the machine.

    • Retr0id 1 hour ago
      > There are countless rare machines that your code doesn't run on--what's so special about big endian?

      One difference is that when your endian-oblivious code runs on a BE system, it can be subtly wrong in a way that's hard to diagnose, which is a whole lot worse than not working at all.

      • electroly 1 hour ago
        That sounds like a problem to deal with as part of your paid IBM s390x porting contract. I guess my point is: why deal with this before IBM is paying you? No other big endian platform matters, and s390x users are 100% large commercial customers. If IBM or one of their customers isn't paying you, there's nobody else who would need it. If IBM is paying you, you can test on a real z/VM that they provide. I see big endian as entirely their burden now; nobody else needs it. If they want it, they can pay for the work.
        • Retr0id 1 hour ago
          I value correct code for purely selfish reasons. The most likely person to try to run my code on a BE system is me.
          • Retr0id 1 minute ago
            Also, endian-correct code is usually semantically clearer, too. For example, if you're reading network-ordered bytes into an int, an unconditional endian swap (which will produce correct results on LE systems but not BE) is less clear than invoking a "network bytes to u32" helper.
          • eesmith 1 hour ago
            There are a lot of odd (by modern standards) machines out there.

            You're also the most likely person to try to run your code on an 18 bit machine.

    • CJefferson 1 hour ago
      You are correct, honestly, I couldn't disagree more with th article. At this point I can't imagine why it's important.

      It's also increasingly hard to test. Particularly when you have large expensive testsuites which run incredibly slowly on this simulated machines.

  • bluGill 32 minutes ago
    For most code it doesn't matter. It matters when you are writing files to be read by something else, or when sending data over a network. So make sure the places where those happen are thin shims that are easy to fix if it doesn't work. (that is done write data from everywhere, put a layer in place for this).
  • eisbaw 1 hour ago
    I did that many years back, but with MIPS and MIPSel: https://youtu.be/BGzJp1ybpHo?si=eY_Br8BalYzKPJMG&t=1130

    presented at Embedded Linux Conf

  • throwaway2027 1 hour ago
    Is there any benefit in edge cases to using big-endian these days?
  • pragmaticviber 2 hours ago
    It's all fun and games until you have to figure out if the endianness bug is in your code or in QEMU's s390x emulation.
    • rurban 2 hours ago
      Haven't found any bug in QEMU's s390x, but lots in endian code.