.Trashes, .fseventsd, and .Spotlight-V100
Merely plugging a removable drive into a mac (when it has write access) makes OS/X think it can take the liberty to write a lot of hidden garbage onto that disk. If you want to stop this from happening, you have to put some special files on that disk before you plug it in.
To stop OS/X from doing Spotlight indexing, you need a file called .metadata_never_index in the root directory of the removable drive.
To stop OS/X from making a .Trashes directory, you need to make your own file that *isn’t* a directory and call it .Trashes
To keep it from doing logging of filesystem events on the drive, you need to make a directory called .fseventsd and inside that folder put a single file named no_log
The contents of these files don’t matter, so you can make them empty files using touch. Even better, you could make it a text file with a link to this post, so that you (or someone else) wandering across the files will know what they’re for.
Apple’s choice to do this is incredibly self-serving and shameful. At bare minimum, hidden files and features like these should be off by default for any non-mac-only filesystem formats. They should only be enabled when the user has been made aware of them.
December 2nd, 2009 at 8:33 pm
I’d like to share with you the story of a botched backup software installation, a lost partition table/boot sector, untold hours spend relearning enough about hard disks, partition tables, FAT filesystems, etc., to locate the start of the second partition (which happened to be close on disk to where an older partition used to live).
Partition table finally reconstructed, I plug it into my Mac. Having clicked on the first file it’s clear there is something still terribly wrong with the drive - it seems I’ve created the partition table to point at one of the old ‘nearby’ filesystems, rather than the most recent one.
Ordinarily, simply updating the start of partition +3mb or so would have been all required, but OS X had kindly created all the files you mentioned above - clobbering to death the second partition image.
Never, ever, ever, use OS X for recovery. These are toy computers for toy people, that do toy things when you least expect it.. like writing to a f**king disk without asking. Lesson learned.
December 5th, 2009 at 12:32 am
Actually, many unix systems will write to a mounted disk automatically, unless you mount the fs with the -noatime flag. I think if you mount the disk read/write, that’s giving the OS permission to write to the disk. If you don’t want it to do that (e.g. Because the fs integrity is damaged) the you need to be sure to mount it read only.
Apple’s focus is rightly on providing the best user experience to the non-technical user. People trying to rescue hosed filesystems or avoid harmless dot-files are a special niche, and a secondary concern at best.
December 11th, 2009 at 2:15 pm
@Jeremy:
It is certainly true that most unix-like operating systems AND WINDOWS will automatically write to a file system. However, file system recovery is a basic operation common to ALL operating systems. If the default behavior of Mac OSX prevents recovery, then it is simply the wrong action.
It really is not hard to wait and see whether the user is trying to perform actions that require those added files to be there.
As a Linux user, I am often called upon to recover data for Windows users. Mac users probably get just as many requests. If the OS trashes the drive before one can perform the recovery, then the above comments about not being acceptable for serious usage are true.
January 5th, 2010 at 11:16 pm
Awesome! Thanks for this post. I have no proof, but I suspect these were screwing up my RAID. Incidentally, I have my RAID drive added to be excluded from both Time Machine and Spotlight, yet these directories are still created and contain loads of data.
January 11th, 2010 at 2:01 am
Folks connecting external drives to a Mac should mount them read-only if they don’t want them written to (just like a savvy user would ‘mount -r’ or ‘mount -o ro’ under Linux). Here’s a prefpane (”ReadOnlyMounter”) that helps with that:
http://homepage3.nifty.com/extant/MacOSXsoft/MacOSXsoft.html
I agree, the use of .Cruftiness to tell Spotlight and fseventsd and Finder not to drop crap on my drives is irritating. I use these to help with that:
http://www.zeroonetwenty.com/blueharvest/
http://myhowto.org/mac-os-x/52-disabling-spotlight-in-os-x-leopard-for-the-removable-drives/
To be honest, I’d rather live with this stuff than with Windows Vista constantly rearranging my desktop icons and resetting my folder views on its whim, because it thinks one file is a photograph, so therefore the whole folder must contain photos and i must want to view them as tiles, even though every other bleeding time i viewed the folder i viewed it as a Detailed List. At least Apple makes it possible to turn it off….
March 19th, 2010 at 10:17 pm
Thanks very much for this info. Extremely helpful and nice to know I can stop annoying things from happening in circumstances where another or specialized system tries to do things with these files that it shouldn’t do.
April 14th, 2010 at 10:40 am
Thanks! I just noticed those files/folders on a flash drive that had I most recently used with a borrowed laptop (Win XP Pro) while on vacation. I thought the laptop had been infected, since it had outdated anti-virus and no firewall. (It came from my girlfriend’s office and she didn’t want me putting on any of the GOOD security tools I use.)
BUT, having read this, I remember that the day before the vacation, I had been in a computer store and had plugged the flash drive into a MacBook on sale, to see if it could transparently read my existing files. So that’s how I got these stupid things! I can stop writing my long e-mail to my girlfriend about how her office laptop may be infected…
(I was checking out the MacBook because with slightly dropping Apple prices and the rare sale, the MacBook was “only” $200 more than comparable WinXP and Win7 notebooks. (Comparable except for the Mac’s dinky 13 inch screen, that is, but I guess that contributes to the long battery life.)
I’m going to put the files you recommend on all my flash and USB drives just in case I ever plug any of them into a Mac again.
Oh, there is one other thing that the MacBook wrote on the flash drive that you didn’t mention in your blog: a file (not folder) in the root called “._.Trashes”.
April 14th, 2010 at 3:32 pm
Oh, there is one other thing that the MacBook wrote on the flash drive that you didn’t mention in your blog: a file (not folder) in the root called “._.Trashes”.
Hi Randy; I didn’t see that particular file in my case. But Googling for “._.Trashes” shows you’re not alone.
As a general concept, the ._ prefix is explained here:
http://www.westwind.com/reference/OS-X/invisibles.html
“These files are created on volumes that don’t natively support full HFS file characteristics (e.g. ufs volumes, Windows fileshares, etc). When a Mac file is copied to such a volume, its data fork is stored under the file’s regular name, and the additional HFS information (resource fork, type & creator codes, etc) is stored in a second file (in AppleDouble format), with a name that starts with “._”. (These files are, of course, invisible as far as OS-X is concerned, but not to other OS’s; this can sometimes be annoying…)”
April 14th, 2010 at 3:57 pm
Thanks. I’ve never really understood why MacOS “speak with forked tongue” — seems an unnecessary complication, like the Windows Registry. (Dear Windows: If a program needs configuration data, store it in a damn file in the program directory!)
I guess the extra “._” files make sense for cross-file-format compatibility though.
May 14th, 2010 at 6:10 am
This is even more annoying when you’re connecting to a remote WebDAV server.
http://code.google.com/p/sabredav/wiki/Finder
August 4th, 2010 at 6:30 am
I need urgent help: my friend has connected my portable external hard drive to your Mac, but Mac did not recognize the external HD.
After that I tried to connect the external HD on my PC (Windows Vista) and the drive no longer appears. I used Windows tools and I can notice that the external HD is connected, but I can´t access any content.
By a data recovery software, I can see the complete original contents in the external HD, but also has in it some new files (to me) as: .Trashes and .Fseventsd.
I believe that by the recovery program data I can copy the contents of the external HD to my PC, but I appreciate help me with:
- Is there any way to undo what the Mac did??
- Also I am not knowing how I’ll be able to access the external HD from my PC (for instance: if necessary to format the external HD) since I can´t see it from PC.
Thanks
August 4th, 2010 at 8:52 pm
If these files are already on the removable drive, can they just be deleted without causing any corruption, or is there a specific process to remove them?
August 30th, 2010 at 7:03 pm
David W:
Why so hostile? Why call names (”toy people”)? If you hate Macs, sell yours and stop the verbal abuse of others.
Hostile Fork:
“Apple’s choice to do this is incredibly self-serving and shameful.” What makes you think there is no use for the user in these hidden files and that they are purely for the benefit (”self-serving”) of Apple, Inc.? Do these files call home with private data that Apple uses to exploit people?
November 10th, 2010 at 6:14 pm
Thanks for posting this. I do a 2 way sync with a USB flash drive on 4 different windows and OSX machines that I use frequently. I was pulling my hair out when I’d get stuck in sync hell with out-of-date .spotlight junk. Very very useful. I wish these things were clearly documented by Apple.
November 22nd, 2010 at 9:19 pm
Randy, the program directory is a terrible place for configuration data - this folder should be read only. The Windows registry is actually much more appropriate, though personally I prefer the ProgramData and AppData folders.
January 28th, 2011 at 8:56 pm
Steve:
Actually I’m with Randy on that point. Windows (since NT) is the only OS to store its configuration in a single monolithic and vulnerable datastore. Whenever possible, I force all my programs to store its settings in its own program folder. This way I have never lost important configuration data and have rarely, if ever, had to actually reinstall anything, except for windows itself and I haven’t reformatted in over ten years.
However, in the current 21st century climate of pandering to the low-browed, formatting-is-a-solution-for-everything and OMG BIG BUTTONS! masses, I wouldn’t advise non-techs to do this.
March 14th, 2011 at 9:55 am
I think it’s unfair to characterize the indexing and logging as “shameful.” Apple’s metadata and filesystem features are powerful and satisfy a lot of use cases, especially for the average user.
That said, I 100% agree that it’s very bad behavior to just set up shop with .Trashes, .Spotlight-V100, and .fseventsd directories on external filesystems that aren’t in Mac OS filesystem format. Many times external devices are very proprietary (digital cameras, voice recorders, etc) and when I throw away a file, I want it gone, not just moved into the .Trashes directory where it still takes up space. This alone drives me nuts; I can’t imagine the frustration of trying to do volume recovery when the OS is scribbling on it.
March 18th, 2011 at 10:40 am
There should be some standard way - maybe in prefs - to force MacOSX not to put this garbage on the drive. Right now you have to manually put FOUR! pieces of your own junk on the drive to prevent OSX from doing it.
For some reason, it doesn’t pollute my SD card when it is in my Samsung camcorder and I attach it. But if I put the same SD card in an adapter, it throws the garbage on, so I know it can do it.
My problem is less with Apple deciding to do this than with leaving no way to turn it off.
March 20th, 2011 at 1:49 am
You guys seem so ignorant. If you don’t know how unix-based operating systems work, then gtfo and go back to Win 95, you ms sheeple.
I’ve known about these files since middle school. Ask yourself if your OS automatically indexes the contents of its volumes, allows you to store trash data without immediately deleting the file (a big reason most people probably come to you for data recovery in the first place), or allows you to store an icon to a volume on the disk itself! These are just some of the “self-serving” features these files enable.
If you suspect a disk to be needing recovery, NEVER plug it in to anything without making sure it won’t write to it. Duh… use a disk utility.
TZ, see the little “lock” switch on your SD card? Didn’t think so.
The majority of you guys just seem to wander into the Mac store and expect them to work like Windows. Go watch Pirates of Silicon Valley and you’ll know why a lot of things are the way it is today, computer wise.
Computers are tools built by people with unique philosophies. Apple’s philosophy is make the customer happy, and never make anything they wouldn’t want themselves. MS’s philosophy, unfortunately, involves making as much money with as little effort as possible.
March 20th, 2011 at 12:29 pm
@Computer Engineering Student (emphasis mine)
Regarding ignorance, there’s a quote attributed to Mark Twain I’d like to share:
“When I was a boy of 14, my father was so ignorant I could hardly stand to have the old man around. But when I got to be 21, I was astonished at how much the old man had learned in seven years.”
March 24th, 2011 at 3:37 pm
“In any disagreement between two or more parties, the truth is always in between them.” And now finally having become intelligent in the eyes of my then 16.5 year old, I will add the following.
The lock switch on SD cards DOES NOT LOCK THE DATA. SD card drive hardware IS NOT REQUIRED to respect this switch’s setting.
And yes, engineers do design according to their personal philosophies. That’s how we get nightmares and a very few fantastic systems. I would submit that Windows, Mac OS (pre-UNIX), and UNIX have all caused nightmares for many. It’s also why DOS is still used for data forensics natively. All other OSes require drive locking hardware between the host and drive.
After some 45 years in engineering, building stuff that several billion people have now used, I’ve learned that Occam’s Razor (see William of Okham) applies in every case. No system should perform any function without necessity. In this case Mac OS should not leave trash behind on a non-Mac file system. If Mac OS really needs additional files or directories, then it should put them on its own system disk. And note that garbage collection on the flash drive itself is not viable, as it requires user cooperation to not rip the drive out before clean-up occurs.
Finally Apple is just as much a pirate as any other in Silicon Valley (my home), Redmond, or anywhere else. The many people who don’t buy Apple products, do so because Apple does not make them happy as customers. Steve Jobs is more a tyrant than Bill Gates ever was, and altruism is not a part of Apple any more than it is at any other corporation. Pandering to incompentent users brings in more dollars, and the IT support industry, one as large as the hardware/software development sector itself, sees sees this too. Not cynicism, this is just reality. As George Bernard Shaw wrote: “The power of accurate observation is commonly called cynicism by those who have not got it”.
April 12th, 2011 at 12:09 pm
Does anybody else think it is funny how touchy some people get when people talk smack about the operating system they use?
May 23rd, 2011 at 5:47 pm
[Ed’s paraphrase of this comment: “I do not agree with the arguments of the person who cited 45 years of experience in engineering.”]
September 22nd, 2011 at 10:00 am
Very useful information. And without entering the flame wars, I merely would have hoped for Apple to create the option to switch it of if needed. I need it switched off sometimes, when creating a lot of sticks to give away.
For the rest: I think it is time to call in Godwins Law: nazi, nazi, nazi.
November 4th, 2011 at 2:47 pm
Nice try on Godwin’s Law. Unfortunately, the law explicitly states that it is not invoked when one uses the term ‘nazi’ to invoke it.
Also, it is shameful because Apple made an assumption that it can just write what it wants to a drive I may be just using to store files. Like 10TB of files. Now it’s full of 230,000 files of this garbage.
The other mystery is why some file dates get changed. I’ve had to resort to mounting these volumes read-only from my desktop mac till I can trust mac to play nice. Especially when the file date is a critical piece of information I’m trying to record.
March 13th, 2012 at 3:11 am
Thank you so much for this blog post. It was certainly beginning to annoy me heaps (and I also think the extra hidden dot files were the reason why the SD card got corrupted data/format and required a total wipe via the camera’s built-in SD card formatter).
April 4th, 2012 at 12:53 pm
Thank you for this (the original) explanation and resolution. One thing I didn’t see mentioned is the amount of HD space used up by .Spotlight-V100 files if one doesn’t realize they are storing detailed information about every file on the computer. I noticed that the free space on my HD was leaking away at a rate of several GB a day and became alarmed (understandably, I think) until I found that this was due to the creation of monster .Spotlight-V100 files. Since I prefer to use familiar, simple tools like “locate” to find files by name (and after 50 years of programming experience I have developed the habit of giving files informative names), I have little use for Spotlight in the first place.
Those who believe Apple would *never* install spyware to harvest useful information from those incredibly detailed analyses of all files on a system must truly have swallowed the Kool-Aid. Like all corporations, Apple makes every decision based on the “bottom line”. The only reason they would pass up this goldmine would be fear of expensive litigation if they get caught. Watch for new legislation authorizing them to spy on you “to protect the children”.
April 18th, 2012 at 3:20 pm
I used my flashdrive in a mac and now i have a ton of these files, which even though they are .whatever my windows computer keeps showing me them and clogging up my view. my linux computer has the courtesy to consider it a hidden folder.
I say mac should die horribly for excessive cruelties, and windows is incompetent. Either way, they’re both screwing us all up.
May 1st, 2012 at 10:37 am
All very technical and good but I use an iMac for day to day computing and a Nikon D70 as a camera. I download the jpeg files to the Apple but find that the .fsevetsd, .trashes and ._.trashes files then render the compact flash useless when reinserted in the Nikon. I end up booting up an old Windows laptop to reformat the card before use - reformatting in the camera does not work and assume reformatting in the Mac would still leave these files behind - lost half a dozen photos yesterday because of this!
Suggestions of alternative methods of overcoming this welcome.