Recover a failed TimeMachine backup

I recently received an unpleasant warning message after TimeMachine routinely tried to perform a backup:

Time Machine completed a verification of your backups on “matmos”. To improve reliability, Time Machine must create a new backup for you.
Click Start New Backup to create a new backup. This will remove your existing backup history. This could take several hours.
Click Back Up Later to be reminded tomorrow. Time Machine won’t perform backups during this time.Time Machine completed a verification of your backups on 'matmos'. To improve reliability, Time Machine must create a new backup for you.  Click Start New Backup to create a new backup. This will remove your existing backup history. This could take several hours.  Click Back Up Later to be reminded tomorrow. Time Machine won’t perform backups during this time.

Googling around for others with the same problem I found quite a few tips (like this one, or this one). The basic idea is to mount the sparsebundle image, run a disk check/repair, and hope for the best. In my case (as you will see in a bit), my sparsebundle appeared to be hosed. My options: lose my old backups or look for a way to recover the old backups. But first up, turn off TimeMachine, and then try to run a standard disk check.

Run disk check/repair

  • Unlock and mount the TimeMachine sparsebundle from the already-mounted server share (of course your server name, network share, sparsebundle names will not be the same as mine):

    $ sudo chflags nouchg /Volumes/TimeMachine-David/fünke.sparsebundle
    $ sudo chflags nouchg /Volumes/TimeMachine-David/fünke.sparsebundle/token
    
    $ sudo hdiutil attach -nomount -noverify -readwrite -noautofsck /Volumes/TimeMachine-David/fünke.sparsebundle
    /dev/disk2          	GUID_partition_scheme          	
    /dev/disk2s1        	EFI                            	
    /dev/disk2s2        	Apple_HFS
    
  • The disk check utility fsck may now be running, so get the PID of it and kill the process so we can manually run it:

    $ ps auxwww | grep fsck
    $ kill PID
    
  • Now run fsck with some repair options on the correct disk partition (use the “Apple_HFS” partition as listed in the mount step above (/dev/disk2s2 in my example):

    $ sudo fsck_hfs -dryf /dev/disk2s2
    journal_replay(/dev/disk2s2) returned 0
    ** /dev/rdisk2s2
    	Using cacheBlockSize=32K cacheTotalBlock=65536 cacheSize=2097152K.
       Executing fsck_hfs (version diskdev_cmds-557~393).
    ** Checking Journaled HFS Plus volume.
       Invalid number of allocation blocks
    (4294967295, 0)
    	IVChk - volume header total allocation blocks is greater than device size 
    	volume allocation block count 102374400 device allocation block count 97630464 
    ** The volume could not be verified completely.
    	volume check failed with error 7 
    	volume type is pure HFS+ 
    	primary MDB is at block 0 0x00 
    	alternate MDB is at block 0 0x00 
    	primary VHB is at block 2 0x02 
    	alternate VHB is at block 781043710 0x2e8dc7fe 
    	sector size = 512 0x200 
    	VolumeObject flags = 0x07 
    	total sectors for volume = 781043712 0x2e8dc800 
    	total sectors for embedded volume = 0 0x00 
    CheckHFS returned -1317, fsmodified = 1
    
  • The disk check did not do anything, so let’s unmount the sparsebundle:

    $ sudo hdiutil detach /dev/disk2s2
    
  • You can try running the disk check (fsck) multiple times. Some have reported that does the trick! In my case, it didn’t help. I had to try something else.

What to do next?

So, “The volume could not be verified completely” says that disk check is not going to repair my sparsebundle. But one good thing to note: the sparsebundle can be mounted read-only, so the old backups should still be there. So the plan: make a new sparsebundle, mount it, mount the old sparsebundle, and then copy all files from the old sparsebundle into the new. Sounds easy, right?
Making a new sparsebundle is not rocket science, however copying the files from TimeMachine backups can be quite challenging. I learned quite a bit when trying various methods to copy the files across (Finder, cp, rsync, ditto, etc). TimeMachine is quite ingenious: it uses a combination of file hard links and directory hard links (the latter is a new one to me!) in order to keep the backup size at a mininum. Unfortunately, all the methods I tried could not reconcile the directory hard links: instead of the links being created, the actual directory contents were copied. Furthermore, Apple has made it difficult to work directly with files in TimeMachine backups by making use of sandboxing and an access control “safety net” (see this or this). So I did some more digging and found a great product that can deal with TimeMachine backups, directory hard links, and this safety net: SuperDuper.

Recover contents of failed sparsebundle

Move failed sparsebundle to a new location

I tried restoring the failed sparsebundle to a new sparsebundle while both were on a network drive (connected to my machine via gigabit ethernet) and the backup was painfully slow (wasn’t finished after 24 hours) meaning the bottleneck was random access/seek time of my poor, slow network drives. I cancelled the restore operation, moved the failed sparsebundle to an external USB drive.

Create new sparsebundle

With the failed sparsebundle no longer in my TimeMachine network share, I created a new sparsebundle. Since I encrypt my backups, I used TimeMachine to create a new sparsebundle on the network share:

  • Enable TimeMachine, select the network share, select “encrypt backups”, then “use disk”
    select network share in TimeMachine
  • Provide encryption password:
    create sparsebundle encryption key
  • Let the backup run, then cancel after a few minutes. This will create a new sparsebundle on the network share.
    backing up...

Mount both sparsebundles

  • Mount the failed sparsebundle (from the external/USB drive). Unfortunately, you can’t use the paste command in the encryption password field 🙁
    field does not allow pasting!
  • Note that the sparsebundle will be mounted read-only (which is just fine):
    Mac OSX can't repair the disk 'Time Machine Backups'. You can still open or copy files on the disk, but you can't save changes to files on the disk. Back up the disk and reformat it as soon as you can.
  • Now that both sparsebundles are mounted and have the same name (Time Machine Backups), we need to make sure we know which is the source (external drive) and which is the destination (network share). A little commandline magic:

    $ mount
    ...
    /dev/disk4s2 on /Volumes/Time Machine Backups (hfs, local, nodev, nosuid, read-only, mounted by david)
    /dev/disk5s2 on /Volumes/Time Machine Backups 1 (hfs, local, nodev, nosuid, journaled, mounted by david)
    
  • So, “Time Machine Backups” is the source (it’s read-only) and “Time Machine Backups 1” is the destination.

Copy files from the failed sparsebundle to the new sparsebundle

  • Open “SuperDuper!”
    Select copy: “Time Machine Backups”
    to: “Time Machine Backups 1”
    using: “Backup – all files”
    SuperDuper
  • Select “Options…” and choose “Smart Update” (this prevents SuperDuper from reformatting the destination sparsebundle, ask me how i know 😉 )
    Smart Update
  • Advanced options are left as the default:
    tm-3-sd3
  • Start the copy process:
    Start copy
  • Let it run for a long time…
    running
    done

Enable TimeMachine

Enable TimeMachine and start the backups. When the backups first started, the “Oldest backup” date was not listed. But when the backup finished, TimeMachine successfully recognized the oldest backup. Success!
Backing up
done!

Updated 2013-07-06: updated fsck step to not be recursive, can try to run fsck multiple times, thanks to comments!

23 comments
  1. Maybe Time Machine is trying to be too clever, picking up where it left off when, for example, connection to the disk is broken. If so it clearly needs some additional mechanism to notice “I’ve tried this several times and it keeps going wrong, maybe I need to back off to the last successful state”. Perhaps not something to worry about in less critical software, but not difficult to get right when it is really needed … as in backups!

  2. Note that the potentially very time-consuming “-R” option is not needed on the “chflags” command.

    Only the *.sparsebundle directory itself, and the file “token” within it, have the “uchg” flag set, as one see with the “-lO” option to “ls”.

    You might also mention that some forms of corruption can be difficult or impossible for fsck_hfs or Diskwarrior to fix, but if you’re willing to risk throwing away whatever data is still lingering in the FS journal you can try turning off journalling and then running fsck_hfs again (perhaps after detaching and re-attaching the disk and making sure that journalling has remained off). This can be done with the “/System/Library/Filesystems/hfs.fs/hfs.util” program. It’s a good idea to turn journalling back on again after everything’s fixed and working again since it can allow recovery of data being written during a crash or disconnect.

    (I first found information about the journal disabling trick here: https://dmunsie.wordpress.com/code/hacks/)

    1. Great tips, thanks!

  3. This post was quite helpful, although it may be good for you to mention you need to su to root in order to kill the fsck process that gets kicked off automatically by mounting the sparse bundle. Also, from reading other sites, multiple runs of fsck may repair the issue, so don’t give up after the first one… give up when the results of fsck are the same twice in a row, that means it’s done all it’s going to do. I’m still working my issue but already have different output after a second run, when the first told me the disk cannot be repaired.

    1. Excellent tip, thanks! In my case, the multiple fsck runs didn’t help so I had to try another solution, hence copying from one sparsebundle to another (using SuperDuper).

  4. Hey David,
    I’m about a really noob question so forgive me !!!
    I’ve got the verification error… and i was trying to follow your instructions on how to fix it..
    Anyway… i went into sudo and typed the command in like you’ve said… and i get the following…

    Parvezs-MacBook-Pro:~ Silver$ sudo chflags -R nouchg /Volumes/TimeMachine/Parvez’s MacBook Pro.sparsebundle
    >
    Parvezs-MacBook-Pro:~ Silver$
    Parvezs-MacBook-Pro:~ Silver$

    As soon as i hit return i get that > sign with a flashing cursor. I’m not sure anything is happening.

    In fact the same thing happens if i put in a hdiutil command as well…

    Your help would be appreciated..

    Thanks 🙂
    Parvez

    1. Hello,

      This first command does take a while to perform, however perhaps the path (with spaces) is incorrect? Try typing it in as /Volumes/TimeMachine/Par (then press tab) and it should autocomplete. You’ll see a backslash before the space. Otherwise you can quote the path as well: sudo chflags -R nouchg “/Volumes/TimeMachine/Parvez’s MacBook Pro.sparsebundle”

      Also, see the previous comment where Greg mentions the recursive -R flag is not necessary, which make the command:
      $ sudo chflags nouchg “/Volumes/TimeMachine/Parvez’s MacBook Pro.sparsebundle”
      $ sudo chflags -R nouchg “/Volumes/TimeMachine/Parvez’s MacBook Pro.sparsebundle/token”

      Good luck in restoring your backup, and sorry to not have replied sooner!
      ./david

      1. If you have spaces in your file name try using prior to the space e.g
        sudo chflags nouchg “/Volumes/TimeMachine/Parvez’s MacBook Pro.sparsebundle”

        1. True, good tip! However what you posted is incorrect. It is either:
          sudo chflags nouchg "/Volumes/TimeMachine/Parvez's MacBook Pro.sparsebundle"
          or
          sudo chflags nouchg /Volumes/TimeMachine/Parvez's MacBook Pro.sparsebundle

          You either escape the spaces with a backslash and no quotes are required, or you quote the path and don’t escape the spaces.

          Easiest of all is to just use tab-completion in your shell. Start typing the path into the command and press tab which will “autocomplete” the path for you, filling in the remainder of the path based on what you’ve typed already. The shell will at the same time take care of escaping the spaces in the path for you. Google it for setup instructions if tab-completion is not automatically enabled in your shell.

  5. After I type tail -f /var/log/fsck_hfs.log it give me this:

    /dev/rdisk1s2: fsck_hfs run at Tue Oct 22 21:46:24 2013
    /dev/rdisk1s2: ** /dev/rdisk1s2 (NO WRITE)
    /dev/rdisk1s2: Executing fsck_hfs (version diskdev_cmds-557.3.1~5).
    QUICKCHECK ONLY; FILESYSTEM DIRTY

    /dev/rdisk2s2: fsck_hfs run at Fri Oct 25 09:22:46 2013
    /dev/rdisk2s2: ** /dev/rdisk2s2 (NO WRITE)
    /dev/rdisk2s2: Executing fsck_hfs (version diskdev_cmds-557.3.1~5).
    QUICKCHECK ONLY; FILESYSTEM DIRTY

    Then i re-typed tail -f /var/log/fsck_hfs.log twice just because it’s now been 4h and thought it should be finished but nothing happens…

    Any thoughts?Thanks for the help.

    1. Hmm… I did some googling for “fsck_hfs QUICKCHECK ONLY; FILESYSTEM DIRTY” and found this page, with this response: https://discussions.apple.com/message/17925819#17925819

      As mentioned in the link above, try running the chflag command again, this time recursively on the entire sparsebundle rather than just the sparsebundle and token, which will take considerably longer but hopefully will help:

      sudo chflags -R nouchg /Volumes/TimeMachine-David/fünke.sparsebundle
      

      Good luck!

  6. Only me again !!! I had another failure !!!
    This time… i did the sudo fsck_hfs -dryf /dev/disk5s2 command and it worked. It found an error in the b-tree catalog, and it appears to have repaired it.
    Trouble is… that when i switch on my TimeMachine again on my MacBook… i still get the “complete a verification…” message. I’m not sure what to do next…should i still go through the process of moving the sparse bundle, creating a new backup, stopping it, and then using superduper ?
    Or is their a cleverer way of getting past the error ?
    As always… appreciate the help…
    Thanks

    1. Check out this solution which describes what to do if the fsck_hfs successfully repairs your sparsebundle: you have to modify a plist that records the state of the backup so TimeMachine will use the sparsebundle correctly.
      Good luck!

  7. I have this problem.
    I half deleted sparsebundle and after copy to another dysk get error:

    iMac-Sebastian:~ sebastian$ sudo fsck_hfs -drfy /dev/disk5
    journal_replay(/dev/disk5) returned 0
    ** /dev/rdisk5
    Using cacheBlockSize=32K cacheTotalBlock=65536 cacheSize=2097152K.
    Executing fsck_hfs (version diskdev_cmds-557.3.1~5).
    Block 2 is not an MDB or Volume Header
    Block 1949330782 is not an MDB or Volume Header
    volumeType is 0
    0000: 0000 0000 0000 0000 0000 0000 0000 0000 |…………….|
    . . .
    01f0: 0000 0000 0000 0000 0000 0000 0000 0000 |…………….|
    unknown volume type
    primary MDB is at block 0 0x00
    alternate MDB is at block 0 0x00
    primary VHB is at block 0 0x00
    alternate VHB is at block 0 0x00
    sector size = 512 0x200
    VolumeObject flags = 0x01
    total sectors for volume = 1949330784 0x74306d60
    total sectors for embedded volume = 0 0x00
    CheckForClean – unknown volume type
    CheckHFS returned 6, fsmodified = 0

    1. Hmmm… did you copy the sparsebundle itself, or did you copy the contents WITHIN the sparsebundle (using the program SuperDuper! for example)? If you just copied the sparsebundle from one to location to another, then the copy will also have the same errors as the original.

  8. I appreciate, result in I found exactly what I was having a look for.
    You’ve ended my 4 day long hunt! God Bless you man. Have a
    great day. Bye

  9. Hello there,

    I’m in the middle of performing these steps. One question: when you say “moved the failed sparse bundle to an external USB drive”, how do you do it? My sparsebundle sits on one of the disk on my NAS, and neither the web interface nor the CLI seems to allow me to do so.

    I’ve mounted the remote Time Machine Backup share on one of my other Mac’s, and used the CLI to do a “cp -a”. What do you think? The sparsebundle has been copied correctly so far.

    Thanks

    1. To copy the sparsebundle to an external drive do as you’ve done: mount the network share containing the sparsebundle, then either copy the sparsebundle with CLI or just do it with Finder. Make sure you copy recursively if using the CLI: cp -Ra
      Good luck!

  10. Any ideas what to do when you attach the sparse bundle using hdiutil but it only finds the base device (e.g. /dev/disk3) and not the other two devices (e.g. /dev/disk3s1 and /dev/disk3s2)? Is my backup beyond repair?


    $ sudo hdiutil attach -verbose -nomount -noverify -readwrite -noautofsck /Volumes/Time\ Machine/hera.sparsebundle
    Initializing…
    CBSDBackingStore::newProbe directory, not a valid image file.
    DIBackingStoreInstantiatorProbe: interface 0, score -1000, CBSDBackingStore
    DIBackingStoreInstantiatorProbe: interface 1, score 1000, CBundleBackingStore
    DIBackingStoreInstantiatorProbe: interface 2, score -1000, CRAMBackingStore
    DIBackingStoreInstantiatorProbe: interface 3, score -1000, CCarbonBackingStore
    DIBackingStoreInstantiatorProbe: interface 4, score -1000, CDevBackingStore
    DIBackingStoreInstantiatorProbe: interface 5, score -1000, CCURLBackingStore
    DIBackingStoreInstantiatorProbe: interface 6, score -1000, CVectoredBackingStore
    DIBackingStoreInstantiatorProbe: interface 0, score 100, CBSDBackingStore
    DIBackingStoreInstantiatorProbe: interface 1, score -1000, CBundleBackingStore
    DIBackingStoreInstantiatorProbe: interface 2, score -1000, CRAMBackingStore
    DIBackingStoreInstantiatorProbe: interface 3, score 100, CCarbonBackingStore
    DIBackingStoreInstantiatorProbe: interface 4, score -1000, CDevBackingStore
    DIBackingStoreInstantiatorProbe: interface 5, score -1000, CCURLBackingStore
    DIBackingStoreInstantiatorProbe: interface 6, score -1000, CVectoredBackingStore
    CBSDBackingStore::newProbe directory, not a valid image file.
    DIBackingStoreInstantiatorProbe: interface 0, score -1000, CBSDBackingStore
    DIBackingStoreInstantiatorProbe: interface 1, score 1000, CBundleBackingStore
    DIBackingStoreInstantiatorProbe: interface 2, score -1000, CRAMBackingStore
    DIBackingStoreInstantiatorProbe: interface 3, score -1000, CCarbonBackingStore
    DIBackingStoreInstantiatorProbe: interface 4, score -1000, CDevBackingStore
    DIBackingStoreInstantiatorProbe: interface 5, score -1000, CCURLBackingStore
    DIBackingStoreInstantiatorProbe: interface 6, score -1000, CVectoredBackingStore
    DIBackingStoreInstantiatorProbe: interface 0, score 100, CBSDBackingStore
    DIBackingStoreInstantiatorProbe: interface 1, score -1000, CBundleBackingStore
    DIBackingStoreInstantiatorProbe: interface 2, score -1000, CRAMBackingStore
    DIBackingStoreInstantiatorProbe: interface 3, score 100, CCarbonBackingStore
    DIBackingStoreInstantiatorProbe: interface 4, score -1000, CDevBackingStore
    DIBackingStoreInstantiatorProbe: interface 5, score -1000, CCURLBackingStore
    DIBackingStoreInstantiatorProbe: interface 6, score -1000, CVectoredBackingStore
    CCachedBackingStore::readDataFork returning 29
    DIFileEncodingInstantiatorProbe: interface 0, score -1000, CMacBinaryEncoding
    CCachedBackingStore::readDataFork returning 29
    DIFileEncodingInstantiatorProbe: interface 1, score -1000, CAppleSingleEncoding
    CCachedBackingStore::readDataFork returning 29
    DIFileEncodingInstantiatorProbe: interface 2, score -1000, CEncryptedEncoding
    DIFileEncodingInstantiatorProbe: interface 0, score -1000, CMacBinaryEncoding
    DIFileEncodingInstantiatorProbe: interface 1, score -1000, CAppleSingleEncoding
    DIFileEncodingInstantiatorProbe: interface 2, score -1000, CEncryptedEncoding
    DIFileEncodingInstantiatorProbe: interface 0, score -1000, CUDIFEncoding
    DIFileEncodingInstantiatorProbe: interface 0, score -1000, CSegmentedNDIFEncoding
    DIFileEncodingInstantiatorProbe: interface 1, score -1000, CSegmentedUDIFEncoding
    DIFileEncodingInstantiatorProbe: interface 2, score -1000, CSegmentedUDIFRawEncoding
    DIDiskImageInstantiatorProbe: interface 0, score -1000, CUDIFDiskImage
    DIDiskImageInstantiatorProbe: interface 1, score 1000, CSparseBundleDiskImage
    DIDiskImageInstantiatorProbe: interface 2, score -100, CSparseDiskImage
    DIDiskImageInstantiatorProbe: interface 3, score 0, CRawDiskImage
    DIDiskImageInstantiatorProbe: interface 4, score 0, CDARTDiskImage
    DIDiskImageInstantiatorProbe: interface 5, score 0, CDiskCopy42DiskImage
    DIDiskImageInstantiatorProbe: interface 6, score 0, CNDIFDiskImage
    DIDiskImageInstantiatorProbe: interface 8, score -100, CShadowedDiskImage
    DIDiskImageInstantiatorProbe: interface 9, score -1000, CCFPlugInDiskImage
    DIDiskImageInstantiatorProbe: interface 10, score -100, CWrappedDiskImage
    DIDiskImageNewWithBackingStore: CSparseBundleDiskImage
    DIDiskImageNewWithBackingStore: instantiator returned 0
    Attaching…
    DI_kextWaitQuiet: about to call IOServiceWaitQuiet...
    DI_kextWaitQuiet: IOServiceWaitQuiet took 0.000002 seconds
    2014-07-15 20:24:41.531 diskimages-helper[948:4223] DIHelperHDID serveImage: attaching drive
    {
    "auto-fsck" = 0;
    autodiskmount = 0;
    "hdiagent-drive-identifier" = "92392737-9412-4A9F-B75C-42F281D995DF";
    "unmount-timeout" = 0;
    }
    2014-07-15 20:24:41.534 diskimages-helper[948:4223] DIHelperHDID serveImage: connecting to myDrive 0x4A0F
    2014-07-15 20:24:41.534 diskimages-helper[948:4223] DIHelperHDID serveImage: register _readBuffer 0x10d62c000
    2014-07-15 20:24:41.534 diskimages-helper[948:4223] DIHelperHDID serveImage: activating drive port 18699
    2014-07-15 20:24:41.534 diskimages-helper[948:4223] DIHelperHDID serveImage: set cache enabled=TRUE returned FAILURE.
    2014-07-15 20:24:41.534 diskimages-helper[948:4223] DIHelperHDID serveImage: set on IO thread=TRUE returned SUCCESS.
    Finishing…
    Finishing…
    /dev/disk3

    1. Hi–Sorry it took me so long to see this, maybe you’ve solved it by now? Hmm, not sure what to suggest–maybe the backup really is hosed 🙁

      1. I was able to recover my photos using PhotoRec. I never got the sparse bundle image to mount properly but could treat it as a raw disk.

        1. PhotoRec looks like a great tool, thanks for the tip!

  11. Oh what a shit – this tips-and-tricks completly failed for me:
    I also hat an error returned by fsck_hfs with my Time Machine backup:

    ** Checking Journaled HFS Plus volume.
    Invalid extent entry
    ** The volume could not be verified completely.
    volume check failed with error 7

    So first I’ve tried to move the sparsebundle to another directory on the same storage. For that I mounted the NFS drive by clicking with the finder on the Shared device. Within the Terminal I renamed the sparsebundle by:
    “sudo mv /Volumes//.sparsebundle /Volumes//defect.sparsebundle”

    But with the next step of starting a fresh backup to create a new sparsebundle, Time Machine simply purged my old sparsebundle, even if it was on another name – so now all my old backus are gone.
    Maybe it was a bad idea to keep the extension “.sparsebundle” visible, be aware!!!!!

    Luckily I don’t have any data loss on my Mac machine, I “only” lost all old backups.

Leave a Reply

Your email address will not be published. Required fields are marked *