?

Log in

No account? Create an account
[process] Backing up your fiction like a pro - Lakeshore — LiveJournal
An author of no particular popularity

Jay Lake
Date: 2009-02-09 05:39
Subject: [process] Backing up your fiction like a pro
Security: Public
Tags:process, tech, writing
Nobody has asked this question, but I'm going to answer it anyway: "Jay, what is your data backup strategy?"

Given that I'm in the midst of drafting a novel right now, this is of immediate and daily interest to me, I'm glad you asked!

I have been using word processors since 1985. I have had all the usual sorts of failures -- corrupt hard drives, video blackouts, dropped equipment, water damage. Once, in a hotel room, a picture fell off the wall and smashed my laptop where it sat on the desk. I've even had computers catch fire while I was using them. (Twice.) I was a PageMaker 1.0 user and QuarkXpress 1.1 user, which means I have lost cumulatively hundreds of hours of effort to unexpected application quits and corrupted files.

In other words, I never trust my computer hardware and software to remain safe, stable and accessible from one minute to the next. And since those files on that hard drive contain the words which are literally my stock in trade, I take very good care of them.

First of all, I have my Mac's hard drive formatted into three volumes. One is /sys, for the OS and application set. I store none of my writing or personal data there, and having the separate partition for system files only allows me to occasionally perform stupid computer tricks which can be quite helpful. One is /writing, which in the days of CD burning drives I limited to 600MB so I could burn the entire volume to a single CD-R. Now I've raised the size to 4GB, which I can fit on a single DVD-R. I also have a third volume for everything else — photos, music, miscellaneous files, things which don't need the backup rigor of the writing data. That way I'm not trying to juggle dozens of gigabytes of data in the high-demand version of my backup process, just the writing files.

Second, to guard against file corruption, every time I complete a novel work session or a short story draft — and sometimes work session there as well, if it's a long piece of short fiction requiring multiple writing session — I make a copy of file at the operating system level. In other words, duplicate it rather than "Save as..." within my word processor. This creates a trail of files which may be 100 deep for a novel project. Each file captures the work at that time, which gives me a backtrail if I need to roll back a revision or writing effort to some previous state. More importantly, if my file has begun to corrupt somewhere, and several of the most recent copies are flaky, I can backtrack to the last good file.

By the same token, I use version control numbers on my files. That means when a piece of work is finished, it becomes "Story 1_0.doc". As opposed to "Story 0_1.doc" and "Story 0_1 copy.doc" and "Story 0_1 copy 2.doc" and so forth. When I begin to revise it, the file becomes "Story 1_1.doc", and begins to generate its own trail of duplicates.

Third, I send the copy of the most recent work session (not the master file) to myself via gmail, as well as to a trusted friend in another city. Gmail means the file is stored as an attachment on Google's servers, as opposed to, say, emailing to myself via a POP account and Eudora, which would have the net effect of storing the file right back on my hard drive. Now if my entire computer goes sideways, and I do not have access to other backup mechanisms, I can at least manually retrieve the work by logging into gmail from any computer, or having my friend mail my file(s) back to me.

Fourth, if I am at home, I run Time Machine backups automagically. I have a 1TB desktop drive and a 500MB portable drive which I swap out daily, so both are carrying the most recent Time Machine backup.

Fifth, when I travel, I take the 500MB portable with me, and run Time Machine at least daily on the road. I also take it when I write at a coffee house or a write-in, or anywhere away from my home office setup.

Sixth, when I am away from home and do not have network access (on an airplane, for example), I also copy the file duplicate mentioned above onto a thumb drive which I store in the portable Time Machine drive's carrying case.

That's my second-greatest point of vulnerability, by the way. In the event of an airplane crash, for example, I would probably be forced to abandon both my computer and my backup drive to evacuate. My plan in that situation is to stuff the thumb drive in my pocket and hope I don't have to swim for it. (And yes, I've thought even that through.)

Finally, every two or three weeks, I burn my entire /writing volume to a DVD-R and mail to my aunt in Tennessee. That guards against a regional catastrophe such as a massive earthquake, tsunami or volcanic eruption, ensuring that my data will still be available even if my entire city has burned down. (And yes, I've also thought that through.)

My greatest point of vulnerability? Forgetfulness. Friday I was writing with camillealexa in a coffee house on Hawthorne, and I'd forgotten to bring my portable Time Machine volume with the thumb drive in its case. It had been out on my desk at home for a routine backup, and I'd simply not paid attention when I left. As I was wrapping up my work session on The Heart of the Beast, I went to gmail it to myself and my friend when the network simply bagged out. The file wouldn't attach. And I didn't have my thumb drive.

I felt as if I'd shit on my hands and couldn't find a washroom. I literally itched. It bothered the hell out of me. What if the drive corrupted on the trip home and I couldn't get back into the data? What if I had an automobile accident and the computer was physically damaged? I was stuck for it, so I went home. As soon as I got there, I rectified the problem, but I was extremely conscious of the problem the whole drive home.

Other things I do to protect my data:

Save my work often. I don't use "Autosave" type features because I find them deeply intrusive, but I've trained myself to hit the save key command reflexively. That protects me from an abrupt application exit, though I still might face file corruption issues.

On long work sessions, stop in the middle, make a duplicate file, and send it out. Then resume writing. This also helps protect me from file corruption issues.

Never, ever, ever leave my laptop in a vulnerable location if I can possibly help it. This means an ironclad rule about never leaving it in the car, not even locked in the trunk. As it happens, I drive a convertible, so I never leave anything in the passenger compartment anyway. Even junk, like an empty box, might look valuable to a meth-head or a vandal. I could get my top cut ($2000+ to replace) for the sake of a worthless piece of cardboard. Likewise, I never lock my car for the same reason. But even in the trunk, someone with a lock jimmy tool or a crowbar could have it out; or they could just steal the whole car.

I can replace the car a whole lot easier than I can replace the computer.

Occasionally when I travel I am forced to leave my laptop in a rental car. This makes me very uncomfortable, but that's when I just have to trust that my layered backups are sufficient.

One thing I don't do to protect my data that others recommmend:

I don't keep printouts of everything. I used to do that, years ago, but keeping up with printouts of my work is like the sorceror's apprentice keeping up with the mops and buckets. I simply don't have the space, time or paper budget to do this. Even without saving drafts and interim files as printouts, I should probably keep printouts of finished material, but even then I'd have a stack of paper at least ten thousand sheets tall.

I'm also certain there are a dozen avenues of vulnerability that I haven't addressed. I'm no business continuity guru or data security expert. When I find them, or have them pointed out to me, I modify my backup strategy accordingly.

The words are the most valuable material thing in my life. Due to my work style and choice of tools, I keep them in an inherently unstable container — electronic files on my computer. So I do everything I can to keep them very, very safe.

Originally published at jlake.com.

Post A Comment | 42 Comments | | Link






manmela
User: manmela
Date: 2009-02-09 14:06 (UTC)
Subject: (no subject)
Very interesting. I was mailing out some stories at the weekend at had an issue where I wasn't sure which copy was the latest version. And worse still I had several copies of the same document in different formats (DOC, RTF, etc.) So I'm planning on doing some sort of version numbering to account for both format and the version of the prose.

I work over several machines. The macbook is the main writing machine but the latest story I've worked on is on my PC. I try to put everything into Google Documents, so I have an offsite copy. I've also just started carrying a USB stick with me in the past few weeks but haven't put everything on there.

Whilst it would be annoying to loose a work in progress, especially at the rate I write, most of my planning is cerebral and I always think I can write better, so losing stuff would be more annoying than catastrophic

Although given your current insane throughput, if you did lose a novel in progress, you could probably re write it in a couple of hours ;-)
Reply | Thread | Link



Jay Lake
User: jaylake
Date: 2009-02-09 14:29 (UTC)
Subject: (no subject)
Good lord, man, losing a novel would be catastrophic!

And thing I hate hate hate is redoing something I just did. This is a character flaw I've had to work very hard to overcome in order to become an effect reviser of my own work.
Reply | Parent | Thread | Link



(no subject) - (Anonymous)
Jay Lake
User: jaylake
Date: 2009-02-09 14:30 (UTC)
Subject: (no subject)
Very much evolved over the years. Almost every step above is in response to a lesson learned at some point, usually the hard way.

(And of course, to our mutual benefit.)

I figure any problem big enough to wipe out my data here in Oregon as well as the DVDs in Tennessee will be on the order of an ELE anyway, so publishing won't be so much of an issue afterwards.
Reply | Parent | Thread | Link | Expand



Gary Emenitove
User: garyomaha
Date: 2009-02-09 14:42 (UTC)
Subject: (no subject)
Two responses:

(1) Speaking as one who dated a real OCD patient for awhile, and have minor such tendencies myself, I'm wondering if some of what you described (such as considering a car accident) kinda sorta borders on that, rather than merely professional concerns?

(2) "...emailing to myself via a POP account and Eudora..." Hmmm. I use Eudora, and I thought I was the only one I know who does (it was strongly recommended to me in the 90s and I've stayed with it). Does this mean you use Eudora or did you just happen to pull that name out for the example?
Reply | Thread | Link



Edward Greaves
User: temporus
Date: 2009-02-09 14:49 (UTC)
Subject: (no subject)
It's not normal to consider whether your data is safe from theft, vehicular collision, or simple entropy affecting hardware?
Reply | Parent | Thread | Link



Edward Greaves
User: temporus
Date: 2009-02-09 14:46 (UTC)
Subject: (no subject)
A comprehensive plan. I'm impressed with your thoroughness. I'm also glad to see I'm not the only person paranoid enough to consider my options in regards to things like: cars, planes, buildings burning down, etc.

Though I admit, you defintely have me beat; I would never have considered my whole town going up in flames. (Though frankly, one of the "joys" of living in NJ, is that major catastrophes are relatively rare.)
Reply | Thread | Link



Jay Lake
User: jaylake
Date: 2009-02-09 14:53 (UTC)
Subject: (no subject)
I can point to two active volcanoes from almost anywhere in Portland; from a high place on a good day, two more.

'nuff said.

Edited at 2009-02-09 02:54 pm (UTC)
Reply | Parent | Thread | Link



torreybird
User: torreybird
Date: 2009-02-09 15:07 (UTC)
Subject: (no subject)
I'm so glad you shared this, not least because the silver locket that holds my mini-SD card encased in silicone putty as my next-to-final 4th-layer backup has been made fun of by Some People.

Thanks!
Reply | Thread | Link



(no subject) - (Anonymous) Expand
(no subject) - (Anonymous) Expand
User: ext_137244
Date: 2009-02-09 15:29 (UTC)
Subject: (no subject)
I use subversion to manage all of my files. I back my subversion repository up weekly and save it on a flash drive. I also save a copy of it with Google. I find it's one of the easiest ways to manage docs and revisions. I love creating DIFFS of my work just to see the improvements.
Reply | Thread | Link



misofuhni: Kitty is
User: misofuhni
Date: 2009-02-09 15:43 (UTC)
Subject: (no subject)
Keyword:Kitty is
Wow, that seems really extensive. I'm such a technophobe, most of this was over my head (a bit). One thing I do have to comment on because it gave me a giggle. Typo--automagically. It's in your fourth powerpoint. For some reason, I want to believe that it isn't a typo! (-:

For your dire, cataclysmic, end-of-existence scenarios: For the airplane--have a body pouch on your person (they usually strap around the chest). They are a secure pouch, usually meant for passports. Place the thumb-drive in a Ziploc (sorry for branding) baggie and place in the pouch. If you can't find a pouch, your underwear will do. Use assorted body parts to secure the thumb-drive.

For the natural disaster scenario--do you think you're going to survive any of that to retrieve your work? Then again, post-humus works do very well on the market...

I'll go back to my corner now...
Reply | Thread | Link



User: renatus
Date: 2009-02-09 16:00 (UTC)
Subject: (no subject)
It might not be a typo; automagically is not uncommon usage when referring to these hey-presto technology wossnames. I personally consider it a technical term. *g*
Reply | Parent | Thread | Link



Elizabeth Coleman
User: criada
Date: 2009-02-09 16:00 (UTC)
Subject: (no subject)
Damn right about not trusting Autosave. I love Scrivener, but you can either have a whole scene in the sidebar selected, or you can be in the text itself. Make sure you know which is active. I once deleted an entire scene instead of a single sentence, and then Autosave made sure it stayed that way. (And yes, I back up my work constantly, but I had to re-edit that entire scene again.)
Reply | Thread | Link



User: renatus
Date: 2009-02-09 16:07 (UTC)
Subject: (no subject)
It's interesting to me seeing who finds this paranoid or too much and who doesn't. All of this is the sort of thing good IT people talk until they're blue in the face about when they try to convince users to backup important data--they don't mean burn it to a cd or carry a copy on your thumb drive, they mean redundant backups across different media, and lots of them.

Your method doesn't seem paranoid to me at all, which I say as a former IT person and someone who's seen too many friends lose years of work because their one backup failed or was corrupted when they lost their master copy. It's sensible and makes good use of the technology without needing tons of storage space, and I imagine once a person is in the routine of it, it doesn't take very long.
Reply | Thread | Link



Jay Lake
User: jaylake
Date: 2009-02-09 16:13 (UTC)
Subject: (no subject)
The single hardest step of the process for me is to remember to burn the DVD for mailout. That's because it's essentially an exception behavior. All the other stuff is embedded in my workflow.
Reply | Parent | Thread | Link | Expand



Zachary Spector
User: blackmonkeymage
Date: 2009-02-09 20:09 (UTC)
Subject: (no subject)
Wow, and I thought I was pretty hardcore when I set up a cron job to upload incremental backups to my webhost every day.
Reply | Thread | Link



Zhaneel
User: zhaneel69
Date: 2009-02-09 21:57 (UTC)
Subject: (no subject)
I use SVN via my boyfriend's server. He introduced it to me from his Programming days. I have revisions, history, branches and a repository.

I have a local copy on my HD, and the server copy, which is backed up by the Company that maintains the server. IIRC, the change long is sent to his Gmail account and much can be reconstructed from that, should both of the company backups fail.

I can pull the files from any computer connected to the internet.

When I was writing regularly I also had a Thumb drive for current projects in addition to the SVN.

It isn't as extensive as yours, but the offsite backups through it I think work pretty well.
Reply | Thread | Link



Jay Lake
User: jaylake
Date: 2009-02-10 05:34 (UTC)
Subject: (no subject)
I just want it to be a little cheaper on $/GB...
Reply | Parent | Thread | Link | Expand



browse
my journal
links
January 2014
2012 appearances