“It took months! It was so frustrating I don’t know if I can ever do it again. But I have so many back list books I want to self publish...”
I’ve heard variations on that plaintive theme many, many times. Writers want to bring their back list back to life only to discover that a) They do not have a digital copy; b) the original manuscript is a mess of markup and it’s not the edited, final version anyway; c) they can’t find anyone to restore the book/s for them that fits within their budget. So they do it themselves. Because, seriously, how hard can it be?
On the left, a scanned paperback novel; on the right, the conversion via OCR into a Word doc.
Doesn’t look too bad, now does it? The innocent writer sets about restoring the document, beginning at page one, and… disaster. Why? To understand why you have to know what is going on behind the scenes. The scanned document is an image file. (There are some scanners that convert to text during the scan and that saves some steps and the results are very good, but it doesn’t eliminate all the problems.) OCR is optical character recognition, meaning the program looks at a picture and decides what letter or character it might be. Depending on the typefaces used, font sizes, line height, condition of the paper and other factors, conversion can range from nearly pristine to what looks like glyphs etched into an alien spacecraft. And then… you have Word (or just about any word processor). It takes that converted text and does its utmost best to recreate the formatting.
The OCR conversion on crack, er, in a Word doc.
The program works really hard to recreate the formatting, using various fonts, section breaks, tabs, columns, text tables, images, etc. To give you an idea about how hard it works, the screen shot you see above is for a straight text (no illustrations) novel that is 72,664 words long. The file size as it stands is 7,117 KB. Over SEVEN MB! (the text by itself creates a file that is only 408 KB) There is absolutely no way any person in the world can do battle against that mess in a reasonable amount of time. The more you try to fix the formatting, the worse it is going to get.
So, I’m going to show you a way to restore the text–not the document!–that will allow you to create a new document that is readable, workable, and editable with a minimum of fuss and bother. It won’t take months or weeks. It will take only hours or at most a few days.
You followed Step 1: Scan and Convert. Your document file is ready to work on. (I will be using Word because so many people use Word, but the principles apply to any word processor. Adjust as necessary.) The very first thing you MUST do is acquire a decent text editor. I use Notepad++. (It’s a free download, stable, and for our purposes, easy to use.) Go get it now. You can’t do what I’m about to show you without it.
Ready to begin? Open that bloated Word doc. We are going to do three things:
- Tag the paragraphs
- Delete headers and footers
- Tag italics
This will be a bit tedious. (I have looked for a fast Find/Replace that works every time without making things worse, and haven’t found it. So, while this boring, it DOES work every time.) So put on a movie or queue up some music, make a fresh pot of coffee and get comfortable.
What tag to use? It doesn’t matter as long as it is unique. I prefer the little diacritical character under the tilde key ` –I have never ever used it in decades of writing. I don’t even know why it’s on my keyboard, but there it is, conveniently located and it doesn’t require the Shift key. I always run a quick Find/Replace to delete any instances where OCR conversion has put those in the document. (If I ever run into a case where the writer actually used the ` then I can easily put them back in later.)
Start at the bottom, work your way up, ignore the odd things that happens to the formatting as you work.
You will start at the bottom–it’s less crazy making, trust me. Just tag the start of every paragraph. If you reach a header or footer that is in text, highlight and delete it. If Word has turned it into an actual header or footer and it’s grayed out, you can safely ignore it. If Word has turned your chapter headings into images, then you will have to type in new ones. Make sure you tag those, too. You will also want to tag deliberate blank lines such as scene breaks. I always insert `## to indicate a deliberate blank line.
QUICK TIP: If, for whatever reason, your document isn’t displaying paragraph indents or if they are difficult to see, open the original scan and place it side by side on your computer screen. Use the original as your guide to find and tag your paragraphs.
By the time you reach the beginning the document is going to look insane–ten times worse than when you started. IT DOES NOT MATTER. One more step and you will never have to look at this particular document again.
TAG YOUR ITALICS
Word is going to do a mediocre job of restoring your italics. But that’s okay. You can get most of them now and restore the rest later. For now, do a simple Find/Replace.
This will wrap all your italics in tags. Even though you will have to change the html tags before you return the text to a word processor, I use them now to make other searches easier.
A WARNING: You may be tempted to also tag bolded text such as that found in headers or subtitles. Don’t. It’s unnecessary and it will make extra work for you down the line. Tag bolded text ONLY if instances occur within paragraphs. Otherwise, just do the italics. Same applies for such things as underlined text. I’ll give you some tips about those special cases later.
QUICK TIP: Illustrations, photographs and other graphical images are going to disappear in the next step. You can delete them as you find them if you want, but it’s not necessary. I do note them as I find them, though. What I do:` [IMAGE CAPTION "Buster Bigbelly on his famous trick pony, Pal." 1948, photo by J. Somebody, page 134] If the image lacks a caption, I insert `[IMAGE photo of a man on a horse, page 134] The page number refers to the original book. If I intend to recover and use the images in an ebook or print-on-demand, I handle those separately from the text.
Now open the text editor. In Word do a Ctrl-a (Select All). Then Ctrl-c (Copy). Open a new file in the text editor and do a Ctrl-v (paste). Now your entire document is pasted into the text editor.
From 7MB to 408KB in seconds.
I do the majority of clean up in the text editor. Every document is going to be different and have different issues. Most fiction writers aren’t familiar with text editors and it looks funny and distracting and it makes it hard for them to work. Since I can’t possibly in one blog post cover all the many searches that I use, I am going to go with the bare minimum that will get you where you need to be.
Before we restore the paragraphs we are going to add a space to the end of every line. It’s not always necessary, but when it is necessary you will be very sorry you did not do this. So, since it doesn’t hurt even if you don’t need it, do it. From the menu bar select Search>Replace and open the Find/Replace box. In the Find field type \r and in the Replace field insert a blank space \r. Make it look EXACTLY like this:
Click Replace All and now you have an extra space at the end of every line. Now open the Find box and make it look EXACTLY like this:
Now click Replace All. As soon as you do, your ENTIRE text file is going to turn into a single line. There will not be a single paragraph or line break to be seen.
Next, open the Find/Replace box and make it look EXACTLY like this:
IMPORTANT: I used the diacritical mark as my tag. That is what I ask it to search for. If you used a different tag, use that. Do a Replace All and every paragraph you tagged is now restored. Use Find/Replace All to delete all your tags.
You may have missed a few or mis-tagged a few paragraphs. You can find many of them now with this search. Open the Find box and search for this \n[a-z]
Now tell it to Find Next. This will find any instances of paragraphs that begin with a lower case letter. You can fix those paragraphs manually.
Word probably used a bunch of tabs–often within paragraphs for justifying text. You want those gone. Open the Find box. Make sure the “Extended” circle is checked. In the Find field type \t and put a single space in the Replace field. Do a Replace All and all the tabs will be replaced with a space.
QUICK TIP: If you have an “oh shit” moment and have done something you did not want to do, go to Edit>Undo. Notepad++ will let you go back as many steps as you need to.
DASHES: HYPHENS AND EM DASHES
When the print book was produced, the typesetter used a variety of dashes–hyphens, em and en dashes, half-ems, 3/4 ens, etc.–to lay out the text. Words were hyphenated. I have tried turning off hyphenation in the original converted document to decidedly mixed and mostly unpleasant results. My recommendation in that regard is to not bother. Here is another of those tedious chores that requires the human eye and some common sense. You can use the Find/Replace function to help you along. Scroll through, find an instance of a dash or hyphenation, then select it with any spaces around it and search for other occurrences. You can use Replace to delete unwanted hyphenation, but be careful about using Replace All. Under Edit in the menu bar you will find the Character Panel. It contains all the ASCII characters, including such useful items as em and en dashes. You can insert them manually or use them in the Replace field.
QUICK TIP: While you are fixing the dashes, you will notice all sorts of interesting characters–what I affectionately call “bugshit”. These are OCR artifacts. You might see bullet symbols or British pound symbols or plus signs. You can delete them as you go, or do Find/Replace All to delete them en masse. Just copy/paste the character into the Find field. I highly recommend that if you do a Replace All, that you replace inappropriate characters with a blank space. It’s a lot easier to delete blank spaces than it is to root out joined words.
A WORD OF WISDOM: Relax. This is a tedious process and imprecise. If you obsess about perfection at this point you will drive yourself nuts. Don’t bother going through the text word by word, line by line or even paragraph by paragraph. This is a BIG CLEAN. Suppose your car was wrecked and you took it to the body shop, and what if the first thing the mechanic did was whip out the wax and buffing wheel and start polishing the hood? Um, no. The first thing you do is pound out the dents and make sure the mechanical parts are in working order. The time for wax and polish is later. Right now, just focus on pounding out dents.
TIDY THE SPECIAL FORMATTING/ITALICS
If for some reason OCR conversion didn’t recognize your italics, you can skip this step. I’ll give you some tips later on how to fix that. If you do have italics, be aware that conversion and Word did a sloppy job of it. Use Find to search for your tags. You can search for either <i> or </i>. Use Find Next and go through the text, deleting any unnecessary tags (such as italicized blank spaces) and tidying the rest. Make sure if you delete either an open or closed tag, that you also delete its corresponding tag. Again, if you happen to see more bugshit while you’re doing this, fix that, too.
When you’re done tidying, use Find/Replace to turn the html tags into something that will not give word processors a case of the vapors. Turn <i> into -STARTI- and </i> into -ENDI-. The hyphens and all caps will help refine your search.
GET RID OF EXTRA SPACES
Use Find/Replace All to rid your text of extra spaces.
- In the Find field insert TWO blank spaces; in the Replace field insert ONE blank space. Click Replace All until it tells you it can find no more.
- Make sure the “extended” circle is highlighted. In the Find field type \n with one blank space after it; in the Replace field type \n with no spaces. Click Replace All until it tells you it can find no more.
- Make sure the “extended” circle is highlighted. In the Find field insert one blank space and \r; in the Replace field type \r with no spaces. Click Replace All until it tells you it can find no more.
Congratulations. Have a drink or a piece of dark chocolate. You deserve it. You have repaired the worst damage to your text. If the text editor is driving you nuts, you can stop using it now. In my next blog post, Part 2 of the Big Clean, I will take you back to Word so you can finish the job in a more comfortable environment. If by chance you are intrigued by the possibilities for some powerhouse searches and find/replace functions to clean up issues specific to your project, ask about them in the comments and let’s see if we can come up with a solution for you.