Source Files Update

So last week I talked about the importance of writers shifting their mindset from thinking “Print Documents” to thinking  “Electronic Files.” Judging by the responses I got, I’d say I’m not the only one concerned with this subject. One of the problems is that the tools we use–namely word processors–are superb for producing printed documents, but frustrating, maddening and over-powered when creating electronic files.

Currently, I’m in the process of creating a booklet/cheat sheet to help fiction writers who are NOT computer programmers to painlessly use the tools they have–and are comfortable using–to create electronic files suitable for e-queries, e-submissions and ebooks. You’d think this would be simple, but it’s not. For one thing, there’s a language problem. I’m a fiction writer, not a programmer. I think in terms of print. Adjusting my way of looking at the subject is difficult and I backslide worse than a dieter at a chocolate fountain. I’ve been talking to William Ockham and he thinks my efforts are on the right track, but kind of sad, too, because he is a computer programmer and he is frustrated by how out-of-synch users and developers are. He is currently working on a program for ebook producers. Essentially he wants a program similar to what blogging sites use. The user selects a “theme” for how he/she wants the ebook to look, then loads in their text and images and the program takes care of all the formatting. If William succeeds, I’ll be his first customer.

In the meantime, I’d thought I’d show you what we’re up against.

First, here is a document provided by a writer for whom I’m producing an ebook. Those little marks you see in the image indicate spaces and paragraph returns.

The writer has created a document suitable for print. Everything is nicely laid out on the screen and without the Show/Hide feature toggled, one would look at it and think, “Beautiful.” Here’s how it looks if I convert this and load it on my Kindle.

Because the writer used paragraph returns to make a new page (instead of inserted page breaks) this is what the start of chapter two looks like:

Now here is the same document with all the extra paragraph returns and spaces removed, and the special formatting and paragraph returns tagged.

Here is what the same document looks like loaded into a text editor for HTML formatting:

Here is what the same document looks like when I upload a cleaned up copy into Scrivener for ebook formatting:

Here is what it looks like if I format it for an ebook in Word (for uploading to Smashwords):

Here is an end product I’m going for (still some tinkering left to do, but you get the idea):

I had thought (foolishly) that all my problems were solved by getting away from Word (except to use its turbo-charged Find/Replace feature to clean up files) and using Scrivener to produce ebooks. It does a great job and it’s relatively painless. Last week I learned there is a problem with one of the books I produced. The problem occurred because Amazon updated older model Kindles and now some people will find that some ebooks–not all!–are compressed and they will have to manually enlarge the screens. Why? I think it’s something Amazon did, but hey, I’m a fiction writer, not a programmer. I do not know why.

Until I do know why and can figure out how to prevent that from happening, I’m now leery of Scrivener. So, damn it, I’m doing what I told myself I really didn’t want to do, and am learning HTML in order to produce ebooks. Come to find out, there are different languages and coding for HTML, too! Ay yi yi. My eyes aren’t red just from the Colorado wildfires, but from spending hours trying to figure this shit out.

It shouldn’t be this hard. Electronic files should not be filled with tiger traps and landmines waiting to turn our lovely words into gibberish marching all over a user’s screen. Those e-queries and e-submissions we labor over until they are perfect should not turn into funhouse mirror words the moment we hit SEND. Creating an ebook should not require a degree in Computer Science and fluency in programming languages. There shouldn’t be different formats requiring different source file layouts and conversion programs and different requirements from every single e-distributor.

Woulda, coulda, shoulda…

The reality is, this Tower of Cyber-Babel and the Facebook Effect (taking something that works fine, and people enjoy using, and tinkering and “fixing” it until everybody loathes it) are what we have to deal with right now. It is more important than ever for writers to change the way they think about the files they create and how they will be used (e-queries, e-submissions, ebooks, print) and adjust the way they use their favored word processors.

Watch this space. The cheat sheet is slowly becoming a reality and I’ll let you know when it’s ready to go.



38 thoughts on “Source Files Update

  1. Calibre has a setting for .mobi conversions that disables compression of the files in your book. The problem is, all these applications we use to convert books were intended for other purposes. Can’t wait for William’s solution. If he wants to kickstart, it I’m there.

    • Ooh, there’s an idea. William is doing this on spec, so has to fit it in where he can. It’s a project well worthy of Kickstarter. I’d jump on it in a heartbeat.

      I’ve been using MobiPocket because that’s what Amazon said to use. It’s stinky and fussy and likes to play little tricks. Pfft. I’m switching over to Calibre this week.

  2. I predict you’ll be pleasantly surprised at the relative ease of conversion with Calibre. The only really tricky thing is finding and deleting the metadata.db file every time you make a new build.

  3. I just bought Scrivener because you said you used it!
    Oh, well. It wasn’t that expensive, and may turn out to be useful, anyway. But yes, I’d definitely buy your cheatsheet – and contribute to William’s Kickstater.

    • Abe, don’t give up on Scrivener yet. William just told me what I probably did wrong.

      “Here’s what I think happened. I’ve seen credible reports of the following scenario.This person has a Kindle Touch. In March, Amazon started rolling out an update that allows the Kindle Touch to use the KF8 format originally developed for the Kindle Fire. There is a side effect. In the KF7 (the old .mobi format), any change to the size of the default (“normal”) font was ignored. Now, only on the Kindle Touch, any font size setting less than 12 point will be interpreted as “use the smallest possible font size on the device”. This is pretty obviously a bug, but I wouldn’t count on Amazon fixing it.

      Check to see if Scrivener is setting the default font for the document to something less than 12 point. I wouldn’t be surprised if it uses 11 point. If this is the problem, just changing that one setting will fix it.”–William Ockham

      So, what that means is, after I open up my new file with my template, I should never, ever use any font size smaller than “normal” ie 12 pt. Amazon may or may not fix it, but in the meantime, I can feel safe about using Scrivener (as long I stop doing bone-headed things. heh).

      • I find this happens with writing problems, too: defining the problem really helps find the solution. In my head, not so much. On paper, much better. The very act of creating a coherent explanation can make the problem obvious – if not for myself, then for someone to help. “It doesn’t work,” isn’t useful.
        I have a Scrivener little problem: the manual says that changes to Preferences will apply to future documents. So I go in, change things to the way I want (such as not having that squiggly red line appear immediately under my mistypings – I really hate that because it yanks me out of writing mode into editing mode and ruins the mood), then save a new project with a new name. Start typing in new project document – squiggly line is there. Fume. Find keyboard control that makes it go away – but it shouldn’t have been there in the first place – I followed instructions. It was a new document, made after the Preferences were set. Sigh.
        For now, I go back to Word. Will do battle in the future.
        Computers: can’t live with them, can’t live without them. But if I had a minute back for every hour I’ve wasted like that…
        PS – Happy to read what I’ve done wrong if someone knows how I messed up.
        PPS – Word does similar stuff – I’m not blaming Scrivener – but I’m used to it in Word.

      • You should go report this over at the Literature & Latte forums. Those guys are always really quick to respond, and they would probably want to know about this for a future bug fix.

        Anytime I’ve had a problem with Scrivener, they’ve not only acknowledged the problem immediately, but they usually found me a work around or fixed Scrivener and gave me a beta the same day. Even when it was my boneheadedness they still turned around the same day and showed me where I screwed up. 🙂

  4. That little squiggly line is my friend, Abe, given that I’m the world’s crappiest typist.

    Anyhow, have you tried creating your own template and applying your preferences to it? When you open up a Blank Template it has its own default settings. If you make a template, your preferences should become the default. Worth a shot.

    • Creating a balnk template – which I know works for word – and setting that to the default template, ought to work. But I have only just started using Scrivener, as in I just put stuff into it last week and should probably go through the tutorial again. :}

      Now Word – there’s a program I’ve been using since 1999 to create both technical and personal documents.

      To turn off the Red squiggling in Word 2007, click on the Windows button (that big icon thing in the top left) and go to Word Options – Proofing and find “Check spelling as you type” – unckeck the box and voila! No more Red Squiggles as you type. In older version of Word the same sort of thing should be found under Tools – Options, Spelling and Grammar. It’s been a while since I’ve used anything but 2007, so I’d have to double check that at home on the kids’ computer.

      Like Jaye, I’m also a horribel typist, as well as a messy speller, so I tend to leave the red squiggles on. I do turn them off for NaNo – makes it more like writing in a notebook. *grins*

      :} Cathryn

      • I have a funny relationship with spellcheckers. i appreciate the squiggly line–craptastic typist–but I’ve never actually used the spellchecker. If i get a squiggle I suspect is an actual misspelling, i pull out my dictionary.

      • I generally try to correct the spelling myself before resorting to the spell check suggestions. Even then I look at the word, close the box and do the changing myself. My spelling has greatly improved… Sadly I think my typing has gotten worse. :}

  5. The incredibly tiny font problem appears on the basic ($79, non-Touch, non-key, non-3G) Kindle that just got its version 4.1 upgrade, too. So far, for three of four .azw3 files I’ve obtained.

  6. Unfortunately, this is a pretty normal stage along the path of new computer technology. I think Apple is the only company to ever really avoid it. Give it a year or two and there will be a half dozen tools (and Scrivener will be more reliable).

  7. Isn’t Apple kind of a “closed” system? I can see the advantages in it, though it shuts out non-Apple device users. Personally, I think Scrivener is reliable right now. It was user error that tripped me up. The user error was caused by exactly what I keep talking about here–a paper frame of mind. My default is to believe what I see on the screen when I should be always taking in to account what is going on behind the screen. Scrivener is NOT a word processor, but it’s still a program and it’s not going to save me from myself. Heh.

  8. The real important part to making sure your content appears good is the CSS (the stylesheet applied to the HTML). For instance, the font-size for normal paragraph content should be 1.0em (a relative measurement) rather than a fixed measurement (12pt, 12px, etc.) This ensures that the text is scaled to the user’s settings rather than forcing it to some value. Title should be about 2.0em and chapter headings about 1.5em. Yes, this is geek stuff, but there really is no workaround to making a high-quality eBook other than learning HTML/CSS and EPUB. Although, I’m curious to see if William can pull it off.

    • I personally like the fine control I get from being able to tweak my own CSS, but I’m all over William’s idea, mainly for the other 98% of indies and would-be indies who are utterly terrified by geek stuff. When someone gets this right, the revolution will REALLY be underway.

  9. Some of you might be interested in a new tool I’ve been working on – it’s free, open source software and uses text-based formatting codes rather than WYSIWYG (which often translates to “what you see is NOT what you get”, as we’ve all experienced).

    It will import your own prewritten HTML and CSS and pass it through unchanged, if that’s what you need, or you can use the internal text-based markup commands.

    It’s experimental at this stage, so do keep that in mind. 🙂

  10. I think you have discovered your new profession should you ever decide that you need one more thing to do 🙂

    I’m telling you, I’ll be hitting you up for your services when I get to formatting the next novel. You just starting thinking about rates.

  11. I’d like you all to note that Jaye is using my book as an example of formatting issues. Jaye is using it, not me, because my eyes are now crossed. If I try to format anything my eyes immediately cross and my brain begins to ooze out of my ears.
    Jaye, thank you. I can’t believe you can figure these things out.

  12. So, did he get the kickstarter project running? I’ll contribute. I would love a wordpress-like tool to create my future e-book(s?).

  13. Wow, I’m flattered. I’ve been busy with my guest blogging stint over at and didn’t see all these comments. Since there is some interest here, I’ll share what I can of my plans. I firmly believe that writers should use whatever tool works for them. For most people, that’s Microsoft Word. Some folks are using Scrivener and almost everyone else is using some word processor (a flavor of OpenOffice or those WordPerfect holdouts).

    The first thing I’m going to release is a free document to source file converter service (to use Jaye’s terms). You save your manuscript in RTF format (pretty much every program supports RTF) and upload it to my service. My program will go through and do all the stuff that Jaye talks about. It will strip all the formatting except bold, italics, and chapter headings. You get back a nice clean source file in RTF format. You load it up into your tool and save it back as a .doc file and you have a source file suitable as the input for ebook formatting. It’s not much, but it is a nice little timesaver and your ebook formatter will thank you (even if you DIY). Did I mention it would be free?

    I really appreciate all the expressions of support. I hadn’t really given much thought to a Kickstarter, but I am thinking about it now. In the meantime, there is something you could do to help. I need test cases. That is, I need real manuscripts before they’ve been given the Jaye Manus treatment. If anyone has copies of their novels (or short story collections) that they wouldn’t sharing with me, I would really appreciate it. I promise not use them for anything other than perfecting my software. I will send you the cleaned up version and destroy or return the original when I’m done.

    If you can help in this way, save your gnarliest files (smart quotes, em dashes, paragraphs indented with tabs and spaces, whatever) in RTF format and
    email them
    to razoroftruth at
    gmail dot

    Let me know what program (i.e Microsoft Word) and version (like 2000 or 2007) and whether you are using Windows, Mac, or Linux (or other Unix variant).

  14. Here’s how I’ve produced e-books, and I’ve been happy with the results. I use Open Office/Libre Office. The same techniques should work with Word, but I haven’t tried it.

    The trick is that all formating should be done with the styles. You can assign paragraph styles (indent, no-indent, blockquote, etc.) and character style, if you need any specity styling for arbriary words/selections. Example: create a style for ephasized text as well as, internal dialogue and or non-verbal communication. Even if they are all dispalyed as italics, making distinct style is key to re-styling the book as needed. Don’t work too hard at getting the details of your styles to look right, as they probably won’t be preserved. The real point, however, is that when the file is converted to e-book formats, all the text will be appropriately tagged for easy styling. *Note*, use the prebuilt Heading paragraph styles for Chapter Headings, for automatic detection and building of TOC. After that, I use Calibre to covert the ODT file to Epub, then I tweak the epub to edit the CSS file. This part, alas, needs a bit of HTML knowledge. However, even if you need to seek help, it should take less than 10 minutes to tweak the style of the paragraphs (assuming you use less than 5 paragraph style in your typical novel.) In most cases, the only thing I’m compelled to change here is the adding maring-top: 0 and margin-bottom: 0 to the text paragraph styles so there is no spacing between paragraphs.

  15. As I move into self-publishing I’m learning about all the little traps lying in my path. For my first project I am working on creating a master file (in Scrivener as it happens, because it’s such an easy way to organize documents, but it will also be contained in a master folder with all the different files separate). I need to be able to refer back to all the elements I used in the book (images, text and metadata, even research) at a later date.

    As far as I can see, for every project I need one completely clean rtf file (with just bold and italics) as the text master, on which all changes and updates must be made. From that, by using a text editing program, I can produce an html file with the correct tags for quotes, em dashes, ellipses and all the rest. With those two files I should be able to cover every eventuality, from conversion to e-book to uploading a pdf to Bowker.

    For my first project I did use the services of a professional formatter, and I would recommend this to first-timers and those who just don’t want to have to learn the techie stuff, especially if (like me) you want fancy chapter headings and so on.

    But in the long run, a program that would provide complete clarity in converting from that basic rtf to every other type of file you need to produce an ebook or print book would be amazing. My ideal would have a split screen so that you can actually see your source and destination formats side by side, and would keep up with all the latest file formats to incorporate every new feature supported (proper drop caps, anyone?) It would also have to be accepted by the major distributors of ebooks and print books the way they now accept Word, so it would have to be a stable and trusted piece of software.

    Tall order? Yep. But where there’s a need i.e. a market, there’s someone out there to fill it. With all the self-published writers out there, there’s definitely a market so I salute you pioneers and will watch with interest!

    • That’s brilliant, Jane. I’m struggling with mine but I think even I could get the hang of a split screen tool. That way, I could learn as I go and correct my mistakes before I upload something embarrassing.

  16. Pingback: Self-Publishing: Carnival of the Indies Issue #22 — The Book Designer

  17. Pingback: Formatting Ebook . . . Tearing Hair Out | The Passive Voice

  18. I’ve found formatting from any program–including the “easy” conversion programs like Calibre–to be a royal pain. I’m trained up as a web developer with several programming languages under my belt and I still think it’s a pain. The HTML Word spits out makes my eyes bleed. I understand the difficulty; the original file is actually an XML document made specifically for Word to display in all of its fanciness, and it doesn’t translate well into other doc types (including, sometimes, PDFs). I use preset styles for all formatting and wrote my own function in PHP to clean up Word’s “filtered HTML” with regular expressions, e.g. take out all the redundant markup and apply my own reliable CSS, but that’s only the beginning of it.

    After that you have all of the differences in the display on competing e-readers. The Kindle ignores empty paragraphs, so you need to use breaks within paragraphs for a complete empty line. The Nook’s ePub, on the other hand, will leave double the empty space for a break within a paragraph, so you need either a break wrapped in a previous paragraph or a paragraph with a non-breaking space instead. Kobo just makes me want to tear my hair out. They apply CSS on top of the ePub to override the native styles, forcing all paragraphs to be aligned left, for example. I pretty much gave up trying to format anything the way I want to on Kobos, especially since you can’t try local files on the PC version, so your book has to go live before you can test the sample from their cloud.

    As a web developer, the headache is just run-of-the-mill. I’m used to spending hours adjusting a website to display and function consistently on Firefox, IE, Opera, and Chrome/Safari all at once. On the bright side, most readers aren’t that picky, as long as it’s readable, so you can usually get away with just going with the flow and uploading what your word processor spits out.

  19. I’m not a programmer, so I only understood about half of what you said. I truly feel as if I’m trying to learn a foreign language–and resorting a lot to sign language and grunting and pointing. I don’t like using Word at all for ebooks. Too much stuff that can go kablooey. I don’t even want to know what a Word file looks like after it runs through the Smashwords meatgrinder.

    Right now I’m doing the clean up in Word (do like the find and replace) then i just copy the file into Notepad++, then i convert that file into ebooks. So far, so good.

    I’ve seen many fine examples of good looking ebooks, and those are my inspiration. I want mine to look better. I would like the assurance that they are going to function across the board. That is an assurance I don’t have. Le sigh…

  20. Pingback: Friday Fry-Up — Speakeasy

  21. Pingback: Friday Fry-Up

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s