Can I Ever Trust Apple Again?

I am a former Apple Design Award winner. I am a long time user of Apple products (and NeXT before that). I know my way around actual technology. This is a story about how I lost my trust in Apple.

Let me first sketch my setting. Ignore the references to technical things in this story, they are there for other tech geeks, but the story is for everyone.

Sketching my setup and its history

For various reasons — one being that I want to keep my technical instincts alive, something I deem important for my job, but that I do on my own time — I run my own infrastructure. I run my own mail server, for instance. Raw internet is delivered to my door, the rest I do myself more or less. I have been doing that since somewhere in the 1990s (including initially using uucp on a NeXT and Trailblazer modems — nostalgia indeed). I like macOS because it is a blend of Unix power and a good user experience.

When Apple released its server product (Mac OS X Server) I was one of the first using it, when the software alone was still priced at $2000. OS X Server eventually morphed from an integrated approach to a (low-priced) 'app' (actually, a lot of stuff in the internals of macOS was included). The last 'integrated' product was the Mac mini Server which Apple discontinued in 2014. After that, there was the Server.app. It was a time-saving way of running your own server, as Apple produced GUI-based maintenance and regular updates that included of course the security patches you want. I still had to do a bit of technical work, for instance integrating Letsencrypt certificates and automating its use was an interesting challenge.

Then, in 2018 Apple removed many Open Source elements from Server.app. It told us to move to our own Open Source implementations. It produced a half-completed manual on how to migrate. This manual was never completed but abandoned. Over the years, Server.app lost more and more elements, and running it would more and more conflict with other parts of macOS, e.g. if you wanted to share folders from your server to clients in your setup, then using that from the system with Server.app running the Open Directory LDAP would not work.

So, when moving to macOS Mojave, I had to move to Open Source dovecot, postfix, etc. I ended up using MacPorts as it had technically the most solid approach (I tried HomeBrew at the time but they could not do postfix or anything that included deeper system stuff and their security implications (like loosening control on /usr/local) worried me). I still use MacPorts.

I had learned over the decades that Apple (like NeXT before them) combines 'insane levels of attention to details' with 'insane levels of neglect or abandonment'. I was realistic about that, but nothing had prepared me for what happened next.

The Mac mini that could. Not.

Early 2019 I had to replace my ageing iMac. I decided that I might move away from the integrated form factor and move to the just launched most powerful Mac mini (6-core Core i7) with two separate monitors. It quickly turned out that the graphics was performing very poorly. I attempted to use an external GPU, but that turned out to be ignored by most software. It was also rather unreliable, the setup would hang regularly. I returned the eGPU, bought a 6-core 27" iMac, and started to use that one.

The Mac mini was doing nothing, but just when I had switched to the iMac Apple announced that they were (slowly) abandoning macOS Server and would slowly end it as a product, starting with the Open Source parts that I was using. I could keep running my existing Mac mini Server for a while, but at some point I would have to migrate. So, I decided to keep that (overpowered) Mac mini for that.

The Mac mini that could. But unreliably.

During Christmas/New Year's break in 2019 I set about migrating. Server.app would still provide directory services and device management, but mail and web services for instance would move to MacPorts based Open Source installs. This was actually also an improvement, because I could move to much better software, like a superiour rspamd instead of spamassassin, the former of which later enabled me to become DMARC compliant (it is quite important to have your mail server be trusted by the outside world or your users — family — cannot send mail).

When my new server setup was running, I started to experience frequent 'hangs'. A part of the mail software seemed to trigger it, especially when sockets were used. Especially the file system would seem to freeze. Strangely enough, I was the only one in the communities that had these issues. File system repairs did not help. Apple Hardware Test gave the system a clean bill of health. Apple Tech radar/bugreport was useless (has always been useless, it acts like a black hole).

Once in a while I tried to find out what was going on with the software I was using, in the meantime, the fact that it was unreliable caused me a lot of headache. As security is paramount for me, all my systems are encrypted at rest so that if some burglar steals my hardware, they cannot steal my data. But that means my server could not automatically reboot as using FileVault meant it could only reboot when a password is entered on the keyboard. So, I had to organise backup people during holidays, for instance. But hanging did not happen that often. Once every few weeks or so.

Then came the release of macOS Monterey. I had skipped macOS Catalina (as long as I still get security patches, I'm happy to use an older version of macOS). Hoping that the new version would solve my reliability issues, I upgraded.

The result was disastrous, reliability wise. Instead of hanging once every few weeks, it hung every few days, sometimes twice a day. As 'luck' would have it, I worked from home often, so if it hung, I would simply do a hard shutdown, start it up, enter the password and it would run again.

Enter Linux

For years, I had contemplated using Linux for my server needs, but two things blocked me. One was that my 'portable home dir' setup depended on a macOS-only tool (and this is still the case). The other one was that I feared the work load of adding a complete second stack to my setup, doing its maintenance, upgrades, etc..

But the situation had become untenable and I threw in the towel. I bought an Intel NUC, installed the (Linux Debian-based) Proxmox hypervisor, managed the external disk to be encrypted at rest using LUKS, installed Ubuntu on a VM, and set up a second mail server (the one that really should not go down) and other services using containers. It was of course very useful from the engineering skills/insight department, now I have actual experience with Linux and Docker. The IMAP software dovecot comes with a synchronisation setup (sadly, now deprecated, but still operational for now) meaning that I could have two dovecot IMAP services running, one on the Mac, one on the Ubuntu VM — very antifragile — in a high-available setup. So, I added HAproxy to my OPNsense router/firewall. From that moment on, while the macOS version was still unreliable, the whole setup was reliable. You might wonder why I was still using the macOS version: it has a macOS-only functionality that the Linux version doesn't have, so I do still prefer it.

A bit of normality returned.

Aside: no mail was ever lost as I have an external backup SMTP service running somewhere and decent backups. My users hardly ever noticed something, or if they did it was something not that important (like their Duplicati backup running to the S3-compatible MinIO backend I was running not running for a day — note: I've moved everyone to restic now).

And then I found out what the problem was

In December 2023 for the first time I started to get actual error messages on the Mac mini instead of simply freezing. I got 'Volume mismatch' errors. A new problem, it seemed, but finally something concrete to do an internet search on. And then I cam across a whole lot of complaints about my type of Mac mini and above all the 16" Macbook Pro from the same era that encountered this when macOS Monterey had been released. Because it came with a macOS release, everybody of course suspected a macOS problem. But then people found out something different: it turned out the cause was simply faulty memory in their Mac. Apple's of 'Apple Hardware Test' or 'Apple Diagnostics' would not detect this, it would report the memory is fine. But since this is an Intel x86 Mac, it was possible to create a bootable USB with something called memtest86 on it. And this one told the true story: the memory on my Mac mini was broken. And it probably had been broken from the moment I bought it. All my poor experiences, all the days and weeks lost came from poor Apple hardware.

Aside: a Mac mini is essentially a Macbook Pro without screen and keyboard/trackpad and with more connections. The Macbook Pro 16" inch was the top model at the time and apparently shared the (iffy, if not broken) motherboard with the top-model Mac mini.

I can tell you that this discovery came as quite a shock. Yes, the NeXT bus was a bit iffy, but since using a Mac, where in part you really pay top dollar for getting a max quality product, it had never occurred to me to suspect the new hardware I had purchased. In hindsight, the unreliability of the initial use was explained now too. I was quite angry, after all I had lost a lot of sweat and tears over this, always thinking my use and my software was to blame (though operating systems where you do not mess with the kernel in some way shouldn't crash, period). My family had certainly paid a price for all these issues, too. It wasn't fun.

I moved my 'server' stuff to my iMac (which I 'memtested' first of course, just to be certain). And sure: everything worked like a charm. From that moment on, I used the iMac as my permanent primary server.

Aside: basically, I had never received a properly working product. Legally, in The Netherlands, I might try to undo the purchase as soon as I find (out according to a judge I know), but I bought at a no longer existing Apple Reseller, so that is not a route available to me.

So, I contacted Apple

The people on Apple's help line were friendly. The process however was another painful exercise.

When I had explained the problem to the people at AppleCare, they implied that this situation was indeed so bad for me (I never had what I paid for) that something should be done. But first they wanted to confirm my own findings, so could I hand the Mac mini over to them? Sure, I said. But I'm not really interested in a repair, it's not really wise to invest a lot in a machine that will not get updates for a very long time anymore. I rather not have more migrations than strictly necessary. But if you can offer me some compensation that I spend on purchasing a new machine, that would be welcome (and proper, as far as I'm concerned). The AppleCare person wasn't unsympathetic to that idea.

So, I wiped it completely and dropped it off at their repair centre (that is, after Apple first told me to drop it off at the local Apple reseller who told me "no, we don't do Apple incidents, our tech support is independent and us looking at it costs €75").

A few weeks later, I got a call. They had repaired my Mac mini and rejoice, I would not get a bill even with it being out of warranty, because this was an original failure (i.e. they confirmed that this was an original problem by offering the out of warranty repair for free). New memory had been installed.

That was not what I had agreed to nor what I had been led to expect as a possibility. I called AppleCare again. They person told be they were very unhappy about it, but there was nothing they could do. They were allowed to offer me something, but they were ashamed to have to offer it. Curious, I asked what it was. "A free Apple stylus with my next purchase".

That doesn't feel like a proper compensation for not having received a working product in the first place and all that painful experience. That is an insult. Which is why the real human being on the other end of the telephone felt so ashamed.

Apple Silicon makes matters worse

I used to have a very high trust in Apple hardware. I was realistic about Apple's humanity (don't count on it). But now, I am in a fix. Apple has changed its hardware and restrictions so much that there is no independent check possible anymore. Memtest86 doesn't work on Apple Silicon. So I can buy a new Apple computer, but when I get problems with my software, I will constantly think: "yes, but what if it is a hardware issue?". Apple's own diagnostics is ver weak (it gave my broken system a clean bill of health, remember?) and cannot be trusted.

If I buy an M1/2/3 system and I get problems: how do I know my Apple hardware isn't broken? Apple by definition blames your software. It's diagnostics is untrustworthy.

The last straw

That was around April 2024. I have a busy life and doing migrations is not something I easily find time for. But last week (it is now September 2024) I thought: OK, take your loss, the machine has received new memory soldered, it can run macOS Sonoma, move your stuff back on the Mac mini for the next couple of years.

So, I was almost done, it went pretty smoothly actually. But I had to wait for a certain element. The machine was running, not doing much. Then, the next morning: "your system restarted because of a problem".

Say what?

It turned out the machine had panicked during the night while not really doing anything (running headless, but none of my 'server' processes were actually running, it was just macOS with some added users). The problem was apparently in firmware related to Intel graphics.

Conclusion

There are many ways I have noticed reliability of Apple (which at some point when it was still called Mac OS X was such that the system could be up indefinitely, say when I ran Mac OS X 10.4 'Tiger' with OS X Server, its reliability/continuity was legendary.

These days, I find that Apple has apparently been engineering the hell out of everything in an attempt to get maximum performance (cutting it close with memory timings might be a sign), maximum energy efficiency (USB ports not waking up properly and easily might be a sign of that), maximum security (and an unholy mix of conflicting iOS and macOS approaches to permissions and security). I've seen signs of all these. I've seen processes running fine in the background when launched from a shell, but unable to read an external volume when launched from launchd (macOS basic services starter). I've seen Thunderbolt-4 SSDs being perfectly mounted at boot time while an USB SSD was ignored and required logging in and user action, apparently, USB was waking up too late for the mount to happen.

Reliability has been going down for quite a long time now.

Apple is a weird mix of high quality and low quality, of insane level of attention to detail and insane levels of abandon and neglect. Phones that get updates for many years while there also is software that is half-baked or suddenly abandoned. It is all there. And that includes the direct experience of the user, who can experience a superbly thought out user experience, a cutting edge unreliable implementation, and an inhuman approach to customers, all at the same time.

In the meantime, I have lost the trust I had in Apple hardware, and while I will still be using it this is not a nice feeling. And I am pissed. I paid €2000 for something that was broken from the start. Apple knows it. It caused me and my loved ones a lot of misery. And Apple is not doing the right thing. Less surprising, that: corporations are inhumane bureaucracies and often incapable of 'nice' after all, but irritating nonetheless. Not what I call a nice 'user experience'.