Home » Adventures in Software » TrueCrypt + Mac OS X + Advanced Format Drives = True

TrueCrypt + Mac OS X + Advanced Format Drives = True

I bought a new hard drive from newegg.com the other day. It’s a rather nice Seagate Expansion STBV3000100 with a whopping 3 TB of storage (current price: $119). It’s also got a whopping 4 KB large sectors instead of the standard 512 byte sectors. This is apparently called the Advanced Format. It’s a fancy name for what seems like such a small difference, but it is apparently a bigger change than you might think since those 512 byte sectors appears to have been with us since the dawn of hard drives back in 1956!  Talk about staying backwards compatible forever…

But be that as it may, all my computers (Mac, Windows, Linux) seem to handle this change just fine, but TrueCrypt was less happy when I tried to initialize the drive with it. TrueCrypt is a great disk encryption app and I highly recommend it.  I run it on all my computers (Mac, Windows, Linux) and I use it to encrypt most of my external hard drives — especially those I tend to travel with.  Sadly, its update schedule is not especially frequent and the most recent version (7.1a) has been out for quite some time now. It still works great, even under Mac OS X 10.8 Mountain Lion, but it does seem to carry some rather old fashioned ideas about what works and what doesn’t work under OS X. In particular, it’s rather set on the idea that anything but 512 byte sized sectors are ganz verboten when it comes to OS X. Or in other words, unsupported. In fact, it is so sure of this that it won’t even bother trying it out; it’ll just post a big juicy alert saying:

Error: The drive uses a sector size other than 512 bytes.

Due to limitations of components available on your platform, partition/device-hosted volumes cannot be created/used on the drive.

Possible solutions:
– Create a file-hosted volume (container) on the drive.
– Use a drive with 512-byte sectors.
– Use TrueCrypt on another platform.

The error message is very well written and quite informative, but it is also quite wrong. It may have been true once upon a time, but at least not since Mac OS X Snow Leopard came out in 2009 according to Wikipedia’s article on the Advanced Format.  So to try this out, I downloaded the latest source for TrueCrypt 7.1a. I then dug through the source and removed all the 512-byte restrictions that I could find and — after some initial hair pulling to get the build environment right — recompiled.  And lo, it ran!  Not only that, it accepted my AF hard drive and started reading & writing to it without a care in the world. Success! Or at least so I thought…

Merrily, I started copying the files from my old 2 TB drive onto it. The copy process ran all night and would probably have run for another day or two too if I hadn’t noticed something odd. For some reason or another (call me paranoid, I don’t care), I decided to verify a few of the files that I had copied and lo again: They did not match! Grumble, grumble, grumble. I stopped the copy process and looked closer at them. Some of the copied files’ data matched the originals and some didn’t, but there was no rhyme and reason to it.  Most of the the corruption appeared to happen on multiples of 4K byte blocks, but there were a few cases on 512-byte boundaries too, maybe < 10%.  Worse, re-reading the files yielded different results each time! WTF? Weren’t 4K sectors supported under Mac OS X after all or was there some other glaring bug or limitation  that I  had missed?

The next several days was spent deep in the TrueCrypt code trying to find the responsible culprit. It was all somewhat complicated by the fact that the TrueCrypt app spawns several subprocesses that each runs several threads, so it’s somewhat hard to control under the debugger. There’s also the case of the data path, which is slightly convoluted under Mac OS X. In order to present a decrypted hard drive to the OS, TrueCrypt uses a subprocess that will implement a FUSE file system with two files: “control” and “volume.dmg”. These are mounted hidden on /tmp/.truecrypt_aux_mnt1. From what I can tell, “control” holds miscellanous details about the encrypted hard drive for TrueCrypt’s internal use while “volume.dmg” represents the decoded data from the hard drive. But that in turn needs to be mounted as another file system in order to get access to the real files and this is accomplished by TrueCrypt using hdiutil to “attach” the .dmg file, which will create a new /dev/disk pseudo-device, which in turn will get probed and mounted by the standard file system utilities.

I’m sure I’ve lost most if not all of my non-techie readers by now and the rest of you are probably having hard time staying awake, but just for the record, the culprit turned out to be the EncryptionThreadPool that attempts to parallelize the work by dividing up the data to be encrypted or decrypted into as many fragments as there are CPU cores on the host computer and letting as many separate threads process each fragment simultaneously. It did a great job at this, but when dividing the data to be en-/decrypted, it used the wrong size factor which caused the fragments to be overlapping and the threads to be stepping on each other’s toes. What size factor was used? What do you think? Yup, 512 bytes was hardwired into the code instead of using the host drive’s sector size. A leftover from before variable sector sizes got added to TrueCrypt, I’m sure. Simple to fix in any case.

A little copy & paste and a quick recompile yielded a new TrueCrypt.app. And lo, take III! The files now copy with perfect fidelity, each one an identical to its original. The world was right again — and more importantly, I was finally able to actually use my nice new hard drive the way I wanted.  Now, I could have used some other disk encryption software, but none of them is as portable or free as TrueCrypt. I could also have formatted the drive unencrypted and then used a file container with TrueCrypt to hold the encrypted file system, but that would mean going through the file system driver twice each time I want to access a files. Yech. So instead I spent several days digging through random code until I got my favorite tool working just like I wanted it to. I admittedly swore over my own idiocy many times in the process, but I guess it really does pay to be stubborn sometimes…

OK, long story over. Here’s finally a link to the precompiled binary if you’d like to try it yourself too: http://bit.ly/tc71alen. You’ll need to download OSXFUSE and install that too in order to run it. It comes with a patch file in diff format if you want to build it yourself too, see the enclosed ReadMe.txt file for details.

Advertisements

29 thoughts on “TrueCrypt + Mac OS X + Advanced Format Drives = True

  1. Great work and investigation. It seems this issue has been hovering around for a bit and I’ve been regularly digging hoping for some resolution. It’s nice to see someone coming up with an actual solution to the issue. It would be nice though if the Truecrypt guys would implement an official patch and push out and update sometime but their release schedule leaves a little be desired in that respect… Thank you for sharing this.

  2. Thank you so much, Mr. Smart Guy. I had been extremely shocked by the error message first, since that malfunction would have been a reason for me not to use MacOS at all. Now everything works perfectly with your build ! Perfect work ! Should be implemented in the official build, too. Have a good time !

  3. Great work! I’m struggling to build truecrypt from sources and didn’t move much past installing wxwidgets. What did you do in “after some initial hair pulling to get the build environment right ” to make it compile? Thank you.

  4. Thank you, glad you found it useful. I put some short notes on how to build TrueCrypt 7.1a under Mac OS X 10.8 in the ReadMe.txt file, but here’s a longer version from my notes:

    * Download MacOSX10.6.sdk; unpack in /Developer/SDKs
    * Download wxMac-2.8.12; unpack somewhere; let WX_ROOT point to it
    * Download Cryptoki; unpack somewhere; let PKCS11_INC point to it
    * Run “port install nsasm pkgconfig ”
    * Edit truecrypt-7.1a-source/Makefile:
    – Change “MacOSX10.4u” to “MacOSX10.6”
    – Change “–with-macosx-version-min=10.4” to “–with-macosx-version-min=10.6”
    – Change “gcc-4.0” to “gcc” and “g++-4.0” to “g++”
    – Add WX_CONFIGURE_FLAGS += CFLAGS=”-arch i386″ CXXFLAGS=”-arch i386″ CPPFLAGS=”-arch i386″ OBJCFLAGS=”-arch i386″ OBJCXXFLAGS=”-arch i386″ LDFLAGS=”-arch i386″
    * Edit truecrypt-7.1a-source/Main/FatalErrorHandler.cpp:
    – Change “i386/ucontext.h” to “sys/ucontext.h”
    – Change “context->uc_mcontext->ss” to “context->uc_mcontext->__ss.__”
    * make wxbuild
    * make WXSTATIC=1

    I think that’s it, but I may have forgotten some details. It’s been a while and my notes aren’t that super perfect…

    • I applied your diff to the original source code and I see that it already contains the modifications from these notes.
      I’ve made some minor changes to use sdk 10.7 (variables in Makefile) but I am getting an error from wxWidgets:

      wxMac-2.8.12/include/wx/mac/carbon/private.h:1459: error: ‘Cursor’ does not name a type
      wxMac-2.8.12/include/wx/mac/carbon/private.h:1488: error: ‘ClassicCursor’ does not name a type

      Do you know how to solve this?
      I will try now to find sdk 10.6 as it didn’t come with my version of xcode.

      Thanks

  5. Hey, thank you for your efforts, I’m finally able to open my drive, but I cannot open any data, everything crashes, nor am I able to write anything…does anyone have the same issue?
    Greetings

    System: Mac 10.8.5
    Decrypted NTFS Drive

  6. What an excellent job! I can now read and write + encryption between Windows and OS X Mountain Lion using my NTFS formatted 3TB hard drive. I was alarmed at seeing the “….512 bytes” error message after 3 days of drive encrypting, but after a little searching have found your solution. I agree it should be part of the official build. Many thanks.

  7. Doesn’t appear to be working on Mavericks; worked fine on Mountain Lion, but refuses to open on 10.9

    • Installing OS X 10.9 Mavericks deleted /usr/local where TrueCrypt.app expects to find the OSXFUSE libraries. Reinstalling or updating OSXFUSE will put it back. Do that and I think you’ll find that TrueCrypt runs fine again. (You can update it from System Preferences / FUSE for OS X or reinstall from http://osxfuse.github.io/)

      • I actually I just did a clean install: downloaded OSXFUSE 2.6.1 and installed it. Still not working. I’m guessing it does have something to do with OSXFUSE, just not sure what the issue is.

      • Andrew, have you checked Console for any error messages? You can also run /Applications/TrueCrypt.app/Contents/MacOS/TrueCrypt from the command line and see if it prints out anything interesting. At least on my machine, I just updated OSXFUSE to 2.6.1 from its System Preferences panel and that put the needed /usr/local/lib/libosxfuse.dylib back again.

      • I uninstalled OSXFUSE, and TrueCrypt, redownloaded them and installed them again, Got the exact same problem: TrueCrypt by itself works fine, but after copying in the Lenlo patch it just opens and immediately closes.

        Running from terminal gives the message “Operation no permitted” The libosxfuse.dylib is where it should be, I also manually deleted this and then updated OSXFUSE to make sure it replaced the file (It did).

        It is very strange, this little fix really helped me out back when I was on Mountain Lion, but now I can’t open my encrypted external drive because of this 512 bug. I guess I’ll have to go back to Mountain Lion.

    • For anyone else trying in my same position, you should know that i was able to get this working on 10.9, by erasing my disk and reinstalling 10.8, and then simply updating. Not technically a fresh install of Mavericks, but close enough.

      • I faced the same problem. Even in root couldn’t I started this app through terminal.
        Considering recompiling everything myself. But installing gcc will be painful…

  8. Pingback: RobinPel | Raspberry Pi NAS with Truecrypt and a 4 KB sector size external hard drive

  9. Thanks for your great work. What do you think about submitting your solution to the truecrypt site as new version?

  10. Excellent work, thanks for sharing it and for persevering. Works great on Mavericks and has helped me out.

  11. Great job! I experienced some weird problem with truecrypt. I think maybe you can give me some advice. I’m using Mac OS X 10.9 with OSXFUSE 2.6.2. I’ve created 3 virtual drives. When I click the “Dismount All” button, it takes above 10 seconds to dismount all drives, but if I use the command line “/Applications/TrueCrypt.app/Contents/MacOS/TrueCrypt -t –dismount”, they are dismounted immediately. I have no idea why two versions behave so differently? I dug through the source code a lot but didn’t find any difference. Do you know what causes this? Thanks a million!!!

  12. hey there !
    First of all thanks for this great workaround !
    I have a problem with my WD mybook 3tb drive. It mounts, I can read and write without any problem. BUT every morning, it is dismounted. I do not know where to look to identify the error. So my question is, where is the error log of truecrypt located ? Same question for OSXfuse as it is maybe related to it?
    Thanks for your help !
    By the way…I’ve tried to ask the devs of truecrypt when this owuld be officially supported by the app…never got any answer. Is the dev team of TC still alive ? active ? or is the development stopped ?

  13. Hello,
    For me the problem is only partially solved: I can finally use my hard drive trucrypt with OSX but some files are corrupted during transfer: (

  14. Thank you so much, this is a life-saver. Worked fine for me on Mavericks. TrueCrypt has a very slow update cycle…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s