Restore Your Back List Books: Step 2: Part 1: The BIG Clean

It took months! It was so frustrating I don’t know if I can ever do it again. But I have so many back list books I want to self publish...”

I’ve heard variations on that plaintive theme many, many times. Writers want to bring their back list back to life only to discover that a) They do not have a digital copy; b) the original manuscript is a mess of markup and it’s not the edited, final version anyway; c) they can’t find anyone to restore the book/s for them that fits within their budget. So they do it themselves. Because, seriously, how hard can it be?

On the left, a scanned paperback novel; on the right, the conversion via OCR into a Word doc.

On the left, a scanned paperback novel; on the right, the conversion via OCR into a Word doc.

Doesn’t look too bad, now does it? The innocent writer sets about restoring the document, beginning at page one, and… disaster. Why? To understand why you have to know what is going on behind the scenes. The scanned document is an image file. (There are some scanners that convert to text during the scan and that saves some steps and the results are very good, but it doesn’t eliminate all the problems.) OCR is optical character recognition, meaning the program looks at a picture and decides what letter or character it might be. Depending on the typefaces used, font sizes, line height, condition of the paper and other factors, conversion can range from nearly pristine to what looks like glyphs etched into an alien spacecraft. And then… you have Word (or just about any word processor). It takes that converted text and does its utmost best to recreate the formatting.

The OCR conversion on crack, er, in a Word doc.

The OCR conversion on crack, er, in a Word doc.

The program works really hard to recreate the formatting, using various fonts, section breaks, tabs, columns, text tables, images, etc. To give you an idea about how hard it works, the screen shot you see above is for a straight text (no illustrations) novel that is 72,664 words long. The file size as it stands is 7,117 KB. Over SEVEN MB! (the text by itself creates a file that is only 408 KB) There is absolutely no way any person in the world can do battle against that mess in a reasonable amount of time. The more you try to fix the formatting, the worse it is going to get.

So, I’m going to show you a way to restore the text–not the document!–that will allow you to create a new document that is readable, workable, and editable with a minimum of fuss and bother. It won’t take months or weeks. It will take only hours or at most a few days.

You followed Step 1: Scan and Convert. Your document file is ready to work on. (I will be using Word because so many people use Word, but the principles apply to any word processor. Adjust as necessary.) The very first thing you MUST do is acquire a decent text editor. I use Notepad++. (It’s a free download, stable, and for our purposes, easy to use.) Go get it now. You can’t do what I’m about to show you without it.

Ready to begin? Open that bloated Word doc. We are going to do three things:

  1. Tag the paragraphs
  2. Delete headers and footers
  3. Tag italics

This will be a bit tedious. (I have looked for a fast Find/Replace that works every time without making things worse, and haven’t found it. So, while this boring, it DOES work every time.) So put on a movie or queue up some music, make a fresh pot of coffee and get comfortable.

What tag to use? It doesn’t matter as long as it is unique. I prefer the little diacritical character under the tilde key ` –I have never ever used it in decades of writing. I don’t even know why it’s on my keyboard, but there it is, conveniently located and it doesn’t require the Shift key. I always run a quick Find/Replace to delete any instances where OCR conversion has put those in the document. (If I ever run into a case where the writer actually used the ` then I can easily put them back in later.)

Start at the bottom, work your way up, ignore the odd things that happens to the formatting as you work.

Start at the bottom, work your way up, ignore the odd things that happens to the formatting as you work.

You will start at the bottom–it’s less crazy making, trust me. Just tag the start of every paragraph. If you reach a header or footer that is in text, highlight and delete it. If Word has turned it into an actual header or footer and it’s grayed out, you can safely ignore it. If Word has turned your chapter headings into images, then you will have to type in new ones. Make sure you tag those, too. You will also want to tag deliberate blank lines such as scene breaks. I always insert `## to indicate a deliberate blank line.

QUICK TIP: If, for whatever reason, your document isn’t displaying paragraph indents or if they are difficult to see, open the original scan and place it side by side on your computer screen. Use the original as your guide to find and tag your paragraphs.

By the time you reach the beginning the document is going to look insane–ten times worse than when you started. IT DOES NOT MATTER. One more step and you will never have to look at this particular document again.

TAG YOUR ITALICS

Word is going to do a mediocre job of restoring your italics. But that’s okay. You can get most of them now and restore the rest later. For now, do a simple Find/Replace.

RestoreBlog4This will wrap all your italics in tags. Even though you will have to change the html tags before you return the text to a word processor, I use them now to make other searches easier.

A WARNING: You may be tempted to also tag bolded text such as that found in headers or subtitles. Don’t. It’s unnecessary and it will make extra work for you down the line. Tag bolded text ONLY if instances occur within paragraphs. Otherwise, just do the italics. Same applies for such things as underlined text. I’ll give you some tips about those special cases later.

QUICK TIP: Illustrations, photographs and other graphical images are going to disappear in the next step. You can delete them as you find them if you want, but it’s not necessary. I do note them as I find them, though. What I do:` [IMAGE CAPTION “Buster Bigbelly on his famous trick pony, Pal.” 1948, photo by J. Somebody, page 134] If the image lacks a caption, I insert `[IMAGE photo of a man on a horse, page 134] The page number refers to the original book. If I intend to recover and use the images in an ebook or print-on-demand, I handle those separately from the text.

Now open the text editor. In Word do a Ctrl-a (Select All). Then Ctrl-c (Copy). Open a new file in the text editor and do a Ctrl-v (paste). Now your entire document is pasted into the text editor.

From 7MB to 408KB in minutes.

From 7MB to 408KB in seconds.

I do the majority of clean up in the text editor. Every document is going to be different and have different issues. Most fiction writers aren’t familiar with text editors and it looks funny and distracting and it makes it hard for them to work. Since I can’t possibly in one blog post cover all the many searches that I use, I am going to go with the bare minimum that will get you where you need to be.

RESTORE PARAGRAPHS

Before we restore the paragraphs we are going to add a space to the end of every line. It’s not always necessary, but when it is necessary you will be very sorry you did not do this. So, since it doesn’t hurt even if you don’t need it, do it. From the menu bar select Search>Replace and open the Find/Replace box. In the Find field type \r and in the Replace field insert a blank space \r. Make it look EXACTLY like this:

RestoreBlog9Click Replace All and now you have an extra space at the end of every line. Now open the Find box and make it look EXACTLY like this:

RestoreBlog6Now click Replace All. As soon as you do, your ENTIRE text file is going to turn into a single line. There will not be a single paragraph or line break to be seen.

Next, open the Find/Replace box and make it look EXACTLY like this:

RestoreBlog7IMPORTANT: I used the diacritical mark as my tag. That is what I ask it to search for. If you used a different tag, use that. Do a Replace All and every paragraph you tagged is now restored. Use Find/Replace All to delete all your tags.

You may have missed a few or mis-tagged a few paragraphs. You can find many of them now with this search. Open the Find box and search for this \n[a-z]

RestoreBlog8Now tell it to Find Next. This will find any instances of paragraphs that begin with a lower case letter. You can fix those paragraphs manually.

Word probably used a bunch of tabs–often within paragraphs for justifying text. You want those gone. Open the Find box. Make sure the “Extended” circle is checked. In the Find field type \t and put a single space in the Replace field. Do a Replace All and all the tabs will be replaced with a space.

QUICK TIP: If you have an “oh shit” moment and have done something you did not want to do, go to Edit>Undo. Notepad++ will let you go back as many steps as you need to.

DASHES: HYPHENS AND EM DASHES

When the print book was produced, the typesetter used a variety of dashes–hyphens, em and en dashes, half-ems, 3/4 ens, etc.–to lay out the text. Words were hyphenated. I have tried turning off hyphenation in the original converted document to decidedly mixed and mostly unpleasant results. My recommendation in that regard is to not bother. Here is another of those tedious chores that requires the human eye and some common sense. You can use the Find/Replace function to help you along. Scroll through, find an instance of a dash or hyphenation, then select it with any spaces around it and search for other occurrences. You can use Replace to delete unwanted hyphenation, but be careful about using Replace All. Under Edit in the menu bar you will find the Character Panel. It contains all the ASCII characters, including such useful items as em and en dashes. You can insert them manually or use them in the Replace field.

QUICK TIP: While you are fixing the dashes, you will notice all sorts of interesting characters–what I affectionately call “bugshit”. These are OCR artifacts. You might see bullet symbols or British pound symbols or plus signs. You can delete them as you go, or do Find/Replace All to delete them en masse. Just copy/paste the character into the Find field. I highly recommend that if you do a Replace All, that you replace inappropriate characters with a blank space. It’s a lot easier to delete blank spaces than it is to root out joined words.

A WORD OF WISDOM: Relax. This is a tedious process and imprecise. If you obsess about perfection at this point you will drive yourself nuts. Don’t bother going through the text word by word, line by line or even paragraph by paragraph. This is a BIG CLEAN. Suppose your car was wrecked and you took it to the body shop, and what if the first thing the mechanic did was whip out the wax and buffing wheel and start polishing the hood? Um, no. The first thing you do is pound out the dents and make sure the mechanical parts are in working order. The time for wax and polish is later. Right now, just focus on pounding out dents.

TIDY THE SPECIAL FORMATTING/ITALICS

If for some reason OCR conversion didn’t recognize your italics, you can skip this step. I’ll give you some tips later on how to fix that. If you do have italics, be aware that conversion and Word did a sloppy job of it. Use Find to search for your tags. You can search for either <i> or </i>. Use Find Next and go through the text, deleting any unnecessary tags (such as italicized blank spaces) and tidying the rest. Make sure if you delete either an open or closed tag, that you also delete its corresponding tag. Again, if you happen to see more bugshit while you’re doing this, fix that, too.

When you’re done tidying, use Find/Replace to turn the html tags into something that will not give word processors a case of the vapors. Turn <i> into -STARTI- and </i> into -ENDI-. The hyphens and all caps will help refine your search.

GET RID OF EXTRA SPACES

Use Find/Replace All to rid your text of extra spaces.

  • In the Find field insert TWO blank spaces; in the Replace field insert ONE blank space. Click Replace All until it tells you it can find no more.
  • Make sure the “extended” circle is highlighted. In the Find field type \n with one blank space after it; in the Replace field type \n with no spaces. Click Replace All until it tells you it can find no more.
  • Make sure the “extended” circle is highlighted. In the Find field insert one blank space and \r; in the Replace field type \r with no spaces. Click Replace All until it tells you it can find no more.

Congratulations. Have a drink or a piece of dark chocolate. You deserve it. You have repaired the worst damage to your text. If the text editor is driving you nuts, you can stop using it now. In my next blog post, Part 2 of the Big Clean, I will take you back to Word so you can finish the job in a more comfortable environment. If by chance you are intrigued by the possibilities for some powerhouse searches and find/replace functions to clean up issues specific to your project, ask about them in the comments and let’s see if we can come up with a solution for you.

 

 

 

 

 

Advertisements

9 thoughts on “Restore Your Back List Books: Step 2: Part 1: The BIG Clean

  1. …and that is why I leave this for you, the pro , Jaye:-) and can not wait working with you again on my second edition of “The Circles of Life: My Ukrainian Family’s…..”

    regards from Jerusalem:-)

    • Good question. It has to do with the natures of Word and the text editor. That bloated Word file is chock full of formatting, most of which you cannot see unless you were to save it as an html document and take a look at what is going on behind the scenes. In my example, there is 7MB worth of formatting.

      The text editor, on the other hand, has NOTHING going on behind the scenes. No styles, no special formatting. The only thing that will show up when you copy/paste the text from Word into the text editor is the actual text and paragraphs. What you see is exactly what you get and the text editor will not add anything to the text.

      So, if you have italicized words in Word, you will NOT have italicized words in the text editor. Poof, gone. So, tag them so you know where they are and they can be restored. When the original pages are scanned and converted into a Word doc, Word will act, at times, pretty arbitrary about where the paragraphs are supposed to be. There will be extra paragraph returns and sometimes blocks of text will be floating without any hard returns at all, or there might be spaces used or soft returns. So, tag the places where YOU know there is supposed to be a paragraph. Then you can easily get rid of the inappropriate paragraphs and retain your own.

      Tagging can be a bit of a pain, but it’s a whole lot easier than trying to repair what shows up in a Word doc.

    • That’s right. The tag is a search term. I use the little diacritical mark because it’s easy (and when tagging paragraphs in a really loooong novel, easy is best) and because it is unique. Tagging takes a while, but once done, it takes only minutes to do the Find/Replace functions to properly restore everything.

  2. Interesting trick tagging the start of paragraphs. We usually merge broken paragraphs… and your way sounds much faster. Gonna have to borrow (i.e. steal) this technique, LOL.

    • Steal away, Paul. The more people who can do this, the fewer I have to do. 😉

      It takes me less than an hour to tag an 80k novel. For shorter works, say under 40k words, unless the conversion is really gnarly, I dispense with the tagging. It takes less time in that case to search for the broken paragraphs.

  3. Pingback: Restore Your Back List Books: Step 2: Part 2: Create a Workable Document | J W Manus

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s