Calibre, Word and MOBI: A Tale of Three Programs

(Yes, I know, MOBI is not a program, but my blog, my headlines…)

Ever since I started blogging about ebooks, I’ve cautioned people against using Microsoft Word to format their ebooks. Not because Word is a bad program and not because it’s impossible to create ebooks with it. It’s because it’s the not quite right tool. Word’s strength lies in creating print documents or pdfs.

Recently, I’ve been cautioning people to not use Calibre to convert their Word files into MOBI files in order to sell them on Amazon. Not because Calibre is a bad program and not because it’s impossible to create MOBI files with it. It’s because it’s not quite the right tool. Calibre’s strength lies in managing a person’s digital library. It was not created to convert commercial ebook files.

EPUB files are not as troublesome as MOBI files. EPUB is EPUB is EPUB, and while each device has its own special way of rendering the file to fit the platform, the differences between devices aren’t big enough for most people to notice. A single EPUB file will work pretty much the same on a Nook as it does on an iPad.

Calibre is set up for optimum use with EPUB files. If a publisher converts a Word (html) file into an EPUB file using Calibre, then what they see there is pretty close to what a Nook or iPad reader will see.

This is not true with MOBI files. The reason is Amazon. You see, EPUB devices have evolved and changed and upgraded and gone the way all technology goes, ever upward and onward. But the device makers built the newer devices around the existing ebook platform. So an EPUB ebook formatted five years ago will work pretty much the same on a new iPad as it did on a first generation Nook. Amazon went bass-ackwards. They built the new devices then tinkered and recreated entirely new ebook platforms to fit the new devices. So a MOBI file being sold on Amazon isn’t just a MOBI file. It’s also a KF/8 file and an iOS file and an AZW3 file and god knows what else is there. I don’t quite get all the technical stuff. What I do get is that the same ebook can work fine on a Kindle Fire, but go to hell on a Paperwhite and look okay on a Kindle Keyboard and turn into gibberish if an iPad user gets hold of it.

The whys and wherefores don’t matter as much as the fact that a file formatted in a program which is optimal for printing documents and then converted with a program that is at its best with EPUB files, is going to have trouble meeting the very odd demands of Kindles.

(By the way, if you are using Scrivner or InDesign to create your ebooks for sale on Amazon, you will run into the same exact problems because Amazon is constantly tweaking and fiddling with the platform(s) and updating devices and they don’t necessarily share what they’ve done with the rest of the world.)

I realize that none of what I just wrote is going to dissuade people from using Calibre to convert their Word docs into MOBI files to sell on Amazon. I know this because people are using Word because that’s the program they know and love(hate) and they need a way to convert those Word files and Calibre is the shortest distance between A and B.

So instead of wagging my finger and clucking my tongue, I did some research. Question: Is it possible to format a file in Word and convert it with Calibre and create a MOBI file good enough to sell on Amazon? (Here, I make a very clear distinction. If your Nook died and you bought a Kindle, and you want to convert all your Nook books into MOBI files you can load onto your Kindle, Calibre is a great tool. That’s personal use. You expect that the ebook might not work completely right, but that’s okay, at least you have it. You can’t ask your paying customers to accept that standard.)

What I discovered is: Yes, it is possible.

I managed to fix the worst problems I see with Calibre-converted ebooks. I managed to create ebooks that respond properly to all the user preferences in three generations of Kindles (Kindle Keyboard, Paperwhite and Fire). I almost got Calibre to build a toc.ncx (what the user sees in the Go To features on Fires and Paperwhites) the way I want it to. I think with some more tinkering and fiddling around inside the opf file, I can fix that problem. I couldn’t get the cover to display on the bookshelf in my Paperwhite, but that’s kind of a non-issue, since Amazon will handle that when the book is uploaded. (It is only a big deal if a publisher is selling direct.)

Even though the ebooks I created this way aren’t up to my standards, they will respond to user preferences and they will look fine and read fine, and thus, they are good enough for uploading to Amazon.

There is a caveat. If you format your document, save it as an html file and convert it as is with Calibre, your ebook will be broken. It will be a substandard product you should not ask people to pay for. What you have to do first and foremost is format your Word file so it works within Calibre’s parameters, and secondly, you have to fix the html coding in the Word file.

Sound scary? It is, kind of. Word’s html coding is a nightmare, full of mso odd bits that give Kindles the hiccups. The good news is, all you really need to do is remove some very specific lines of code and rearrange a few others.

Since this post is running long and I don’t even have any pretty pictures to enliven it, (plus I have a buttload of Christmas gifts to wrap) I am going to explain how I did it in my next post. It’ll have pictures. In the meantime, if any of you, Dear Readers, have figured this out and feel like sharing in the comments, feel free.

Advertisement

I Don’t Hate Calibre, But…

Judging by the heated storm I roused the last time I criticized Calibre it’s probably a mistake to do so again. What the hell. I have to say it:

DO NOT USE CALIBRE TO CONVERT YOUR EBOOKS FOR THE PURPOSES OF SELLING THEM ON AMAZON.

Yesterday I did a troubleshoot and repair on a writer’s ebook. The EPUB she had converted from Word via Calibre was perfectly fine. It was a valid EPUB, and it displayed the way it should on both Calibre and Adobe Digital Editions. Problems arose when she converted the EPUB into AZW3 and MOBI. The ebook worked when she loaded it onto her Paperwhite Kindle, though it had some disturbing issues. Amazon rejected the file outright.

I took her EPUB and ran it through the Kindle Previewer to see what the problem was. It converted with WARNINGS (never want to see that). The ebook opened, but the button under Devices for Kindle for iOS was greyed out. And the cover didn’t display. When I loaded it on my Paperwhite, some of the user controls were locked. I then went back and looked inside the EPUB. The Calibre conversion had declared font-families–Georgia and Times New Roman–neither of which display on Kindles, but they aren’t ignored either, and hence cause all sorts of interesting little problems. Plus, it had built a cover page.

I’m not a techie person, so I don’t know if I can explain it adequately, but I will try. When a file is uploaded to Amazon it converts the source file into a 3-in-1 ebook. If you’ll compare the size of an EPUB to a Kindle, you’ll notice that the MOBI file is about 3 times bigger than the EPUB. The ebook you buy from Amazon will work across several types of devices: e-ink readers, Fire tablets, and a variety of apps, including iOS for Apple. This is tricky stuff. While Calibre can and does convert your files into the MOBI format or AZW3 format, and the files are suitable for personal use, they aren’t suitable for commercial use because the probability is about 97.3% that an ebook converted through Calibre and then uploaded to Amazon will be broken.

This is all about Amazon. They have proprietary platforms and they are constantly updating and improving and tinkering and it’s difficult for outsiders to keep up. Calibre is a library management tool, not a commercial conversion tool. They can’t be expected to stay ahead of the Amazon updates and bugs. That’s not their purpose.

The fix for the writer’s problem ebook was fairly simple. I removed the font-family declarations, removed the cover page, added a line of code to the metadata in the content.opf so Amazon knew the cover was in the file, resized the cover and ran it through the Kindle Previewer. Voila! Everything worked the way it should. The writer was able to upload the file to Amazon with no problems.

My recommendation to the writer for future projects was this: continue using Calibre to convert her Word files to EPUB files since there appear to be no problems and her ebooks are working fine. But, then to refresh her knowledge about using stylesheets in Word via Mark Coker’s Smashwords Style Guide, and direct load her Word file to Amazon. It’s a bit of a pain since you can’t examine and adjust the file before you start the uploading process, but you can take full advantage of the previewer at Amazon and make sure everything works before you hit the Publish button. That way there won’t be any extraneous items like cover pages and “embedded” fonts to muck up the works.

So seriously, folks, don’t kill the messenger or throw rotten tomatoes. This is just the reality. Calibre is not the right tool for converting files to sell on Amazon. Use the program for what it’s intended–managing your ebook library–and find other means to deal with Amazon.

Guest Post: Karen Myers–No Need For Calibre

When I ran the recent ebook formatting contest, I noticed some similar issues with many of the entries, all of them the result of Calibre conversion. I have no beef with Calibre, it’s a great online reading platform and I use it myself, but it’s a not-quite-right tool for converting Kindle books. One entrant, Karen Myers (Perkunus Press), contacted me about the problems in her ebook.

The flaws weren’t fatal, but Karen was not satisfied with not-quite-right. So I pointed her at Sigil and Paul Salvette’s books, tutorials and website, and now I’ll let her tell you the results:

__________________________

Silly me. I’m an old programmer and I pride myself on trying to get my ebooks “just so”, as if I were writing a piece of code. I want to create worthy offerings to add to humanity’s river of books; at the very least, they should be shiny and well-scrubbed.

So when Jaye offered to judge the formatting of a few books from her blog fans, I hopped right on board, and she was very kind in her review.  But I read with horror things like “squishy line spacing” and links to chapters not working quite as they should, systematically.

I use an EPUB reader and hadn’t seen the book on a Kindle device other than the PC Previewer, so it was useful to see this from the Kindle reader’s perspective, since none of my buyers had complained (yet). Without a Kindle device, I hadn’t realized quite how irritating it was to not properly trigger the “Cover” and “TOC” hard buttons.

Now on the one hand, it wasn’t really broken, but on the other hand, I want perfection in book formatting, and some cosmetic and graphic flourishes. I’m not willing to settle for “good enough,” so Jaye was nice enough to coach me through some of the issues.

(Jaye here: I made a template for Karen based on Paul Salvette’s The Ebook Design and Development Guide)

If you’re content with auto-conversion from EPUB to MOBI or vice versa, or output direct to ebook formats from products like Scrivener, then this is overkill for you and you can stop reading now. But if you want as much control as possible over the results without killing yourself, you might find this approach useful.

Originally I formatted my ebooks in raw HTML using tips from a variety of online recommendations, like Guido Henkel’s. The first book was a real learning curve for me, but after that it wasn’t difficult to just do the same for later books.  Automated search/replace macros took care of things like wrapping lines with <p></p> tags, and so forth.

I took the HTML output, opened it in Calibre, added the cover and converted it to EPUB using Calibre defaults (more or less). I did the same with a separate conversion to MOBI, which required me to maintain a different HTML file because of the way Calibre generates the MOBI TOC. These outputs were what I was uploading to the distributors. The MOBI files produced this way were not ideal, possibly because of AZW vs MOBI choices (in other words, me as a Calibre user, not necessarily Calibre as a tool), and I was left with two files to maintain (EPUB and MOBI) and the Smashwords EPUB as a third file, since my approach wasn’t modular.  So every time I found a typo…

Jaye has taught me a better way… (Sigil)

I poured my HTML file for a book, broken up into chapters, into a Sigil template.  Each part of my book has a separate file: Beginning Blurb, Title Page, Copyright Page, Also-By-This-Author Page, Small TOC, Chap 1… Chap Last, Guide & Name Index, If-You-Liked-This-Book Page, Excerpt-From-Next-Book Page, Author Bio Page, Long TOC.

I have 3 outputs: MOBI, EPUB, Smashwords EPUB. The difference between Mobi and the two EPUBs is that the “stylesheet.css” file is a little different between MOBI and EPUB, and the Cover page is treated differently (EPUBs require an extra step). The difference between my EPUB and my Smashwords EPUB version is that the Copyright Page has different content.

Now, thinking in the long term, I expect that the differences between MOBI output and EPUB output are likely to be persistent, and other devices may come along and generate different optimum stylesheet requirements. So I’m fine with having two different (but very similar) stylesheets which I maintain externally and copy in as needed into the stylesheet.css shell.

Likewise, the fact that Smashwords requires its own ISBN means only that I maintain two different external Copyright Page HTML documents and copy the contents of whichever one I want into the shell in Sigil.

Both of these make use of a simple modular structure.

So, what happens when I finish a new book and want to format it?

PREP

1. I copy the Sigil file (MOBI version) from the previous book and rename it.
2. I create Copyright.HTML files for both the normal and Smashwords copyright pages by copying the ones for the last book, renaming them, and updating the content.
3. I create a new Title page (it’s a graphic) and a new Cover.

MAIN CONTENT (MOBI)

1. I work on the MOBI version first (it’s the master). I copy the text in, chapter by chapter, the front blurb, and the back excerpt.  I run saved searches to wrap the lines with <p></p> tags and to convert special characters to named entities.
2. I update the Also By and If You Like This book pages by hand.
3. I run a saved search to update the Title field on all the HTML pages to the new work, and update the equivalent fields in the TOC.NCX and CONTENT.OPF files.
4. If the book is a little longer or shorter (number of chapters) than the last one, I update the TOC.NCX and CONTENT.OPF files and the HTMLTOC file.
5. I update the metadata in the TOC.NCX and CONTENT.OPF files. This allows me to do some things that either Calibre doesn’t, or I don’t know how to find, such as set a UUID (Unique User ID) for my short stories that don’t have ISBNs, embed book descriptions, add keywords, etc.  There’s a great tool for this that Jaye told me about:

bb ebooks meta-pad generator (courtesy of Paul Salvette, BB Ebooks Thailand)

Run the file through Kindle Previewer (which runs Kindlegen) and check the results.

How much time does this take?  I just updated my entire backlist (3 novels, 5 short stories, 1 story collection) to Sigil–it takes me about an hour for a novel, and 20 minutes for a short story.  Making the short story collection from the already-formatted short story files was truly trivial.

EPUB VARIANT

1. Substitute the content of the EPUB stylesheet into the stylesheet.css.
2. Run the Sigil tool to insert the cover.  (Kindlegen does that a different way for MOBI).
3. For the Smashwords variant, substitute the contents of the Smashwords copyright.html for the default one.

That’s it.  I import the files into Calibre for one last look to make sure they seem healthy, and do a quick scroll through on my EPUB reader. If I find typos, I fix them in the MOBI version (and the Scrivener original) and redo steps (10)-(12).

Why not use Calibre?  I am confused by the various options and clearly, for MOBI conversion, I wasn’t doing it quite right. Also, my original HTML file was one big file with a stylesheet and all chapters together, making modular changes clumsy. Calibre created its own version of the styles it found, and they weren’t always what I expected. It’s a big black box to me, and there were some issues with the results, which may be my fault, not Calibre’s.

Why not use Scrivener?  Like Calibre, you are at the mercy of whatever Scrivener decides to do to instantiate the different conversions. Since the Scrivener text isn’t in HTML, there are all the issues of named entity conversions to deal with, and you have little control over the default styles. The results may be clean, but you can’t do anything special, such as use graphic chapter heads, scene dividers, and so forth, at least not in the Windows version.  Perhaps there’s a way…

Other tools, like Scrivener, will take your word processor input and generate EPUB and MOBI output, but the black box in between what you write and what they produce leaves you at the mercy of the limitations of others, and so your output will remain at best functional vanilla. That’s not a bad thing, but we can do better.

It’s really not that hard to go through the learning curve once.  After that, each new book becomes quite easy.  Your book designers or people like Jaye can help you get started by setting up the first one and explaining how it works.

_____________________________

Jaye again. Thank you, Karen. Now I’d like to add a word about Sigil.

Sigil creates EPUB files. Kindle ebooks are MOBI files converted from EPUB files. So, you can take an EPUB file created in Sigil and convert it with KindleGen into a MOBI file that will work on Kindles.

There’s a gotcha–One size does NOT fit all. EPUB and MOBI handle styling differently. They handle covers differently. If you know what the differences are, you can use Sigil to create ebooks that work exactly how they are supposed to, on EPUB and Mobi. You can control how the devices handle your work, as opposed to being at the mercy of whatever the conversion program decides is best.

I have a special offer for readers of this blog. If you are really serious about creating ebooks that work properly across devices, and you are ready to climb the next step on the learning curve, go buy Paul Salvette’s The Ebook Design and Development Guide. It might freak you out the way it freaked me out when I first read it (I have zero programming experience). If it does, but you’re still motivated, send me an email (jayewmanus at gmail dot com) with a screenshot of your receipt for Paul’s book and I’ll set you up with a template similar to the one I made for Karen.

 

 

Changing Ereader Landscapes: What Can It Mean?

I don’t usually post unless I have reached a conclusion of some sort–even if it is wrong (I trust my readers to set me straight). Now I’m just puzzled. This started when I was trying to figure out yet another bug in my Kindle Keyboard. For those who don’t know, that’s the older Kindle model with the tiny keyboard on the casing. Sturdy, reliable, GREAT battery life, and here recently, full of bugs. This time it affects bold face type. Sometimes it displays, sometimes it doesn’t. And I found an instance where it partially displays.

That same day Jon Westcot sent me an email about Nook ereaders. I couldn’t suss out what was going on, though was able, as usual, to come up with some wild-ass conspiracy theories. Here is what Jon sent:

Here’s an interesting situation (meant in the truest Chinese curse way):

With most of your journal postings, you talk about formatting issues, and most of the conflicts come from the support of the “standard” that the various device developers produce and maintain.

But I have found yet another layer to this mess that I wanted to tell you about. I am currently working towards freeing my Nook Tablet from the restrictive yoke of Barnes and Noble’s operating system and replacing it with a basically fully implemented version of the Android OS. (It’s a long, geeky process, but it’s been informative, frustrating… and even fun. And yes, I know I have a “unique” idea of fun.) As part of that process, I’m trying out various ePub readers.

Oh my freaking God!

What a mess! I first downloaded the Nook for Android application. It can’t even find my library of books because I choose to keep them on the secondary memory card. I can’t even contact B&N support because what I’m doing with my Nook violates its warranty, even though the question has to do with their non-device software.

So, I turned to the Google Apps Store to look for ePub readers. The first one I grabbed, while having a huge number of downloads and a nearly 5-star rating, decided to display everything with a Chinese font!

The next one was okay — at least, it showed readable text! But the formatting was all wrong. It took a lot of wrangling to finally find the setting that lets the user accept the publisher’s formatting defaults! Preposterous!

So, I’m still looking for a good e-reader. Until then, I won’t fully migrate my Nook Tablet to the Android OS. But this whole process really surprised — and disheartened — me. It just never occurred to me that the independently-developed e-readers would be so… crappy. I would have expected just the opposite.

I guess this means that we have more things to worry about than “just” the way device developers create e-book reading software — now we have to worry about the software developers, too.

To which my mind leapt to one truly dark scenario and one big conspiracy theory.

I’ve suspected all along that much of what goes into programming and creating platforms for ereaders is created by people who have little concern–or perhaps little awareness–of end users. Is there any other industry that works this way? Maybe high fashion dress designers.

The industry WANTS us all to move happily into the land of tablets. Tablets are cool, but for reading, the dedicated ereaders are far superior. From the industry point of view, all the ereaders can do is, you know, books. Tablets offer chance after chance after chance to sell the user something.

So no, I don’t honestly think dedicated ereader device makers give a shit about the quality of the books that end up on them. In fact, the crappier the ebooks look, the higher the chance that users will turn to tablets. Case in point, I spent time this morning trying to figure out bugs in the eink Kindles. These are bugs that showed up in the latest update. Now for the longest time my Kindle was perfectly stable. Ever since the Fire came out, every update–and there seem to be a lot lately–makes the eink readers work worse and worse. There is no good reason for that. None.

On a side note, I think the Nook is dead. It might be a great device, but B&N has given up on it. The new ‘owners’ (Google? Microsoft?) don’t care because content is king and they aren’t making money off the content going onto a Nook. The only way they can cash in is to lure Nook users into buying tablets. A really nasty sneaky way to do that is to ensure that cross-device apps DO NOT WORK. I think B&N is going to collapse, too. When it goes, that is basically going to leave Amazon and Apple.

I love my eink, but I suspect the bugs are going to get worse while the Fire gets better. Kobo might have a few years while it worms its way into the international market, but in the end the only dedicated readers are going to be shoved into the obsolete closet and there they will stay.

What a state of affairs, eh?

To which Jon replied:

The fact that B&N doesn’t want people using the NC or NT for anything other than what they intend. But these devices are hackable, no matter how hard B&N tries to lock them down. And they do try; every new release of the OS has blocked previous exploits. But those who hack these devices are brilliant. They figure out ways to get around everything, it seems. And that is a good thing! These are OUR devices that we bought and paid for, and we should be able to use them as we see fit.

What shocked me was… well, it was really two things. First, I was surprised at the number of ePub reader applications I found. Most of them are free, which leads into my second shock — just how bad these applications are! I guess one really does get what one pays for. I don’t think I could write an ePub reader (but it might be fun to try), so I can’t imagine why someone would so do and not charge for it, even a paltry $0.99.

It saddens me to think that the Nook may be going away. I really like mine, but I must admit I have thought about saving up for a tablet, though I would really hate to give up my Nook. I really like the new, larger device B&N released, but I haven’t seen anything about its hackability. I looked at it in-store when it first came out and really liked most of the improvements they made to the OS. I asked the rep at the store if the OS would be upgraded for the older devices and he looked at me like I’d just shot a flaming porcupine from my butt. “That’s old tech,” he muttered in defence. More like in ignorance; there was no reason these devices couldn’t run at least 99% of the new OS’s features. I could have argued it with him, but there wouldn’t have been much point to it.

I do think the dedicated ereader makers (at least Amazon and B&N) do care about how ebooks look on their devices. If they look like crap, that reflects badly on their devices. In my opinion. 😉

To which I replied:

You’d think Amazon and B&N would be more concerned, but the evidence says otherwise. Apple is fanatical about quality control, but I suspect it has more to do with protecting the integrity of their devices. Making attractive ebooks is a side effect.

The rest seem happy to let the producers and consumers duke it out. Meaning, it’s up to the producers to play catch up (if they can) and for consumers to keep buying newer, better, fancier devices. It’s weird to me, but I’m not a business person and don’t pretend to be one.

********************************

So that’s it. Something is going on in the land of ereaders and ebooks, but hell if I have a real clue about what it all might mean. My suspicions are two-fold: One) B&N is circling the drain (my bets are on them showing up in bankruptcy court before the year has ended); Two) Dedicated ereader devices are going the way of the 8-track tape player.

I don’t know for a fact what any of this means for me and thee. I probably wouldn’t even know something was up if I didn’t have three models of Kindles so I could watch as one wildly improves (the Kindle Fire) while the others (eink Paperwhite and Keyboard) slowly degrade with every update. Jon’s experience leads me to suspect that nobody cares enough about the Nook to invest in apps and other support for its users.

Make of this what you will.

How The Kindle Works

I talk a lot about broken ebooks, but judging from the search terms that bring visitors to this site, I suspect many people do not know what I mean. Part of the confusion is because I am actually talking about Kindle books (and I should make myself clearer, sorry). Since I don’t own a Nook (and have never played with one), or an iPad or iPhone or Sony reader or magic toaster, I tend to judge ebooks by what shows up on my Kindle.

I suspect there are a lot of people who don’t own Kindles. Maybe one or two who’ve never even seen a Kindle.

Here is Amazon’s dirty little secret regarding its Kindle. As long the ebook is converted to either mobi or prc, the Kindle can read it. (I convert raw documents all the time since I prefer reading on my Kindle over manhandling reams of paper–I just run a Word doc through MobiPocket creator and load it up. I’ve done that with pdf files, too.) I don’t call those ebooks. A person who isn’t aware of how a Kindle actually works, might be unaware their ebook is broken–after all, Amazon let them upload it and reported no problems.

Now, the wise formatter will convert their ebook using KindleGen and check their work on the Kindle Previewer. Amazon just updated it and it has more options, include some font changing, so it’s more reliable now. To truly make sure the ebook works, it should be loaded on a device and run through its paces. People who produce a mobi file using Scrivener or convert a file in Caliber or MobiPocket or run a Word doc through the onsite converter at Amazon might end up with a broken ebook and not know it. (A kind reader might take pity and send you an email to let you know your book is broken, or they might return it for a refund, or–worst of all–they might decide your ebook is too unpleasant to read and not buy any more of your books.)

Pardon my less than stellar photography–here is what the menus look like on Kindles.

The font sizes are pretty hard to screw up. There was a Kindle-induced bug that shrunk the font, forcing users with older Kindle models to have to greatly increase the font size in order to read. I think that bug was fixed. But I don’t think I’ve ever run into an ebook where the size can’t be changed. The fonts themselves can be changed.

• Keyboard: “regular” “condensed” and “sans serif.
• Paperwhite: Baskerville, Caecilia, Caecilia Condensed, Futura, Helvetica and Palatino
• Fire: Baskerville, Caecilia, Georgia, Palatino, and Helvetica

A common flaw is a locked font (usually in the ugliest choice). After looking at the html in ebooks that have “locked” fonts, I think what is happening is the producer, using a word processor, has defined a font the Kindle doesn’t recognize. So it displays in the closest match. But, since the font is defined, it can’t be changed.

Line spacing is an option on all models of Kindle. It’s a useful one and it’s also a common “break.” When I format a book in html I don’t mess with line spacing. I define the line height so my text isn’t squished, but that’s different than single-space, space-and-a-half and double space. Word has a really nasty habit of inserting a definition for line-spacing into the document that will override the user menu. Sometimes this is deliberate on the producer’s part, sometimes it is inadvertent because that’s just how Word rolls. In any case, it’s undesirable.

Another common problem is when the margins don’t work. In the older Keyboard model the user can set how many words there are on a line (fewest, fewer and default) and on the Paperwhite and Fire they can set the margins to narrow, normal or wide. Breaks tend to happen when a producer using a word processor, Scrivener or InDesign justifies the text. Why this affects the margins, I don’t know, but it does.

The Fire allows the user to change the background color. White, sepia or black (sorry, Paul, but black? Oh, my eyes!). It’s a nifty feature, but there is a drawback.

I apologize for the crappy photo, but if you look very closely at the graphic I circled in red you will see a white box around the graphic. For some odd reason Kindle does not recognize that background is transparent. It’s not a huge issue, but one I hope is soon addressed. Something to keep in mind when using graphic elements in your ebook.

Speaking of graphics… Kindles can be read in landscape mode. The Paperwhite requires an ebook that is specifically coded to be read in landscape mode (such as comic panels or a children’s book–unless, there is some command I am too stupid to figure out and am just missing it) The Keyboard can be changed through the menu and the Fire by turning the device.

Landscape mode can have a significant effect on graphics, especially those that are sized to fit the portrait screen. What I do is size the graphics in percentages so that no matter what size the screen or if the book is being read in landscape mode, it will “shrink” or “expand” to fit the text.

(I was reading a novel that had an interesting block graphic in the header. It looked great in portrait mode, but when I flipped it to landscape suddenly it was just a dumb looking box perched atop the text. Yikes!)

Another common problem is page break failure. The best I can tell (and I’m sure there are those smarter than I who will pop in and set me straight) this is a problem when a producer converts an EPUB file into a mobi file through Caliber. Why Caliber destroys the page breaks is anybody’s guess, but it often does.

So what is the poor ebook producer to do? Especially if you do not have a Kindle on which to test your files? (And this isn’t a slam against people who don’t have Kindles–I don’t have an ereader that uses EPUB files, so I’m playing guess and by golly, too. One advantage with EPUB files is that if my file is validated and I haven’t inserted any weird stuff that could override defaults, I’m fairly certain it will work properly. I’d love it if a Nook owner wrote a guest post about its features and common problems. Any takers?)

• Download the Kindle Previewer and use it. It’s not perfect and you can’t test ALL the device features, but it will give you a far better display than Caliber or even the previewer at Kindle Direct.
• If you use a word processor, Scrivener or InDesign be very careful with your style sheets. Do not justify the text. Leave the line spacing at single-space. Don’t get fancy with your margin settings. What YOU see is NOT what the end user will get.
• Experiment with graphic element sizes and use percentages (when possible) rather than fixed em or pixel sizes.
• Learn html and get away from using not-quite-right for ebooks programs.

So, now you know what I mean when I say “broken” ebooks. EPUB readers, what are the common problems you find?

 

 

Are Ebooks Getting Too Complicated For DIY?

I’m a tad out of sorts this morning. Irked, annoyed, disgruntled… Pick one.

Partly it is because of anxiety. I used Paul Salvette’s guide to create my very first complete ebook (one that doesn’t have to go through a third-party conversion process). Now the author is waiting for it to be published on Amazon. I hate the waiting…

Partly it’s because my son gave me a Kindle Fire for my birthday. Oh, wow, is that thing cool. It’s not something I’d have bought for myself. I’m not a gadget person and it takes me forever to warm up to anything new. But wow, the Fire is very cool. The first thing I did was load the book I just finished onto the Fire to see how it looked. It looks gorgeous. It also looks a lot different than on Larry the Kindle.

I looked at other ebooks in my library. I have books put out by big publishers and indie books, and books that were professionally formatted and books that were DIY. Quality is all over the board. Some of the books that look just fine on Larry look amateurish and not-quite-right on the Fire. It’s because many of the books were produced before the Fire existed. The older formatting platform doesn’t translate so well. The standards are different.

The ebooks are readable. I’m not going to pitch a bitch just because a book I purchased last year won’t let me adjust fonts on the Fire. Nor am I going to ping DIY publishers who’ve formatted a Word file according to Amazon’s guidelines and ended up with an amateurish looking ebook.

I’m irked and annoyed at the devices and the platforms and distributors. Quite frankly, this shit has gotten way too complicated.

It doesn’t help that I read Baldur Bjarnason’s latest post at Futurebook. This part worsened my mood:

However, as I’ve written about before, a large proportion of ebooks published are rubbish. Not in terms of the content (although that’s probably also the case) but in terms of the quality of the file. Ereader platform vendors cannot support the full range of CSS that EPUB2 and EPUB3 require because a substantial number of their catalogue would become unreadable.

Platform vendors are in a position where they couldn’t support standards completely even if they wanted to.

No kidding. For instance, while I was building my most recent project, this is what I had to do. Build the file. Launch the file in my web browser. See how it looks. Figure out why something doesn’t look the way I wanted it to look (All the while knowing that what appears in my browser is only an approximation of what will appear on the ereader). Fix and fiddle, then validate the file to make sure it meets EPUB standards. Check how it looks in Calibre (I don’t have a device that reads EPUB). Again, I know that what I see on my computer screen is not necessarily what a reader will see on a Nook or iPad or whatever. Then, I convert the file into MOBI format and load it onto my Kindle. Do more tweaking. Tweaking and fiddling means having to go through validation again. It means more converting and loading and inspecting. And I haven’t even gone through the Kindle Previewer yet. I want to know how my ebook looks on as many devices as possible. I change font sizes and line spacing and the size of the reading window. It’s time-consuming, it’s frustrating, but the worst part is that even though I’m checking and double-checking with everything I have on hand, it’s still not enough. There is no guarantee that an ebook that renders perfectly on Larry the Kindle (and now the Fire) is going to render properly on other Kindle styles or versions, the Nook, the iPad, the iPhone, an Android, a Sony, a whatever.

As the cat sez:

This is, in a nutshell, a disservice to readers. READERS. Do you hear me Amazon? Barnes & Noble? Kobo? Smashwords? Apple? While you guys indulge in device wars and competing formats while creating compatibility issues, are you thinking about readers at all? You know, the people paying the bills? It’s all well and good to roll out the welcome mat to publishers big and small, traditional and indie, and invite all comers to list their ebooks with you. You get your percentage of sales. When your guidelines and standards are such that it is very, very easy for anybody to make a crappy looking ebook, naturally people are going to follow your guidelines to the letter and end up with crappy looking ebooks.

That’s not right. It’s not fair to readers.

It’s difficult making an ebook that renders properly across all devices. For the self-publisher who has neither the time nor the inclination to learn all ins and outs of formatting to meet different standards, it’s damned near impossible.

That’s unfair to the do-it-yourselfer. It’s unfair to their readers.

What’s the solution? I do not know. I’m not a programmer or a tech-type. I have no idea what goes into creating these devices or how they work. I just want ebooks that respect the material and are a pleasure to read. That is not too much to ask. All this screwing around with fancier devices and increasingly complicated and narrow platforms is making it too damned hard.

Source Files Update

So last week I talked about the importance of writers shifting their mindset from thinking “Print Documents” to thinking  “Electronic Files.” Judging by the responses I got, I’d say I’m not the only one concerned with this subject. One of the problems is that the tools we use–namely word processors–are superb for producing printed documents, but frustrating, maddening and over-powered when creating electronic files.

Currently, I’m in the process of creating a booklet/cheat sheet to help fiction writers who are NOT computer programmers to painlessly use the tools they have–and are comfortable using–to create electronic files suitable for e-queries, e-submissions and ebooks. You’d think this would be simple, but it’s not. For one thing, there’s a language problem. I’m a fiction writer, not a programmer. I think in terms of print. Adjusting my way of looking at the subject is difficult and I backslide worse than a dieter at a chocolate fountain. I’ve been talking to William Ockham and he thinks my efforts are on the right track, but kind of sad, too, because he is a computer programmer and he is frustrated by how out-of-synch users and developers are. He is currently working on a program for ebook producers. Essentially he wants a program similar to what blogging sites use. The user selects a “theme” for how he/she wants the ebook to look, then loads in their text and images and the program takes care of all the formatting. If William succeeds, I’ll be his first customer.

In the meantime, I’d thought I’d show you what we’re up against.

First, here is a document provided by a writer for whom I’m producing an ebook. Those little marks you see in the image indicate spaces and paragraph returns.

The writer has created a document suitable for print. Everything is nicely laid out on the screen and without the Show/Hide feature toggled, one would look at it and think, “Beautiful.” Here’s how it looks if I convert this and load it on my Kindle.

Because the writer used paragraph returns to make a new page (instead of inserted page breaks) this is what the start of chapter two looks like:

Now here is the same document with all the extra paragraph returns and spaces removed, and the special formatting and paragraph returns tagged.

Here is what the same document looks like loaded into a text editor for HTML formatting:

Here is what the same document looks like when I upload a cleaned up copy into Scrivener for ebook formatting:

Here is what it looks like if I format it for an ebook in Word (for uploading to Smashwords):

Here is an end product I’m going for (still some tinkering left to do, but you get the idea):

I had thought (foolishly) that all my problems were solved by getting away from Word (except to use its turbo-charged Find/Replace feature to clean up files) and using Scrivener to produce ebooks. It does a great job and it’s relatively painless. Last week I learned there is a problem with one of the books I produced. The problem occurred because Amazon updated older model Kindles and now some people will find that some ebooks–not all!–are compressed and they will have to manually enlarge the screens. Why? I think it’s something Amazon did, but hey, I’m a fiction writer, not a programmer. I do not know why.

Until I do know why and can figure out how to prevent that from happening, I’m now leery of Scrivener. So, damn it, I’m doing what I told myself I really didn’t want to do, and am learning HTML in order to produce ebooks. Come to find out, there are different languages and coding for HTML, too! Ay yi yi. My eyes aren’t red just from the Colorado wildfires, but from spending hours trying to figure this shit out.

It shouldn’t be this hard. Electronic files should not be filled with tiger traps and landmines waiting to turn our lovely words into gibberish marching all over a user’s screen. Those e-queries and e-submissions we labor over until they are perfect should not turn into funhouse mirror words the moment we hit SEND. Creating an ebook should not require a degree in Computer Science and fluency in programming languages. There shouldn’t be different formats requiring different source file layouts and conversion programs and different requirements from every single e-distributor.

Woulda, coulda, shoulda…

The reality is, this Tower of Cyber-Babel and the Facebook Effect (taking something that works fine, and people enjoy using, and tinkering and “fixing” it until everybody loathes it) are what we have to deal with right now. It is more important than ever for writers to change the way they think about the files they create and how they will be used (e-queries, e-submissions, ebooks, print) and adjust the way they use their favored word processors.

Watch this space. The cheat sheet is slowly becoming a reality and I’ll let you know when it’s ready to go.

 

Producing A Kindle Ebook: Design Choices

When Lawrence Block told me he had written a brand new short story about Martin H. Ehrengraf (The Ehrengraf Settlement) and he wanted to produce a collection with all eleven Ehrengraf stories, I was excited. Not just because there is a new story, but because I had ideas.

Here’s the thing. Lawrence Block is an important writer. No matter the format in which his writing is featured–hard cover, paperback, magazine, ebook–the medium should reflect the quality of the writing. I’ve called the Kindle ebooks print books’ “ugly cousins” because of the gray scale screen, uniform page layout, limited typography and the producer’s inability to control the amount of text readers see on the screen. In an earlier post I mulled over why readers seem to be reading differently on the Kindle and in another why some people don’t think of ebooks as “actual” books. Because of that mulling and thanks to insightful readers, I began to think that maybe “ugly cousin” was wrong, but that instead we’re dealing with an “ugly duckling” and there are swans awaiting to be born.

To turn this ebook into a swan, I had two goals:

• Reader friendly
• Make it look worthy of the material

To make the ebook reader-friendly, I tackled it on two fronts. The first was with the source files. That required squeaky clean files with no extra spaces or hidden codes. This isn’t rocket science, but it requires paying attention to details such as uniform punctuation. I produced clean source files which I then loaded into Scrivener for formatting.

The second front was in arrangement. Mr. Block wrote these stories in order, so it made perfect sense to put them in the order they were written. Since his fans also enjoy his forewords, introductions and afterwords, it also made sense to include those in the book. Because the main character, Martin H. Ehrengraf, is enamored by poetry and often quotes it, each story has an epigraph consisting of poetry or a poetic quote. Here is where I had to make some design decisions. Do I compile the introduction, epigraph and story into a unit? Place the epigraph above or below the chapter head? I decided to split it all up. The introductions are small stories in and of themselves. The epigraphs would serve as “appetizers” giving the reader a visual rest from the story text. Plus, by setting them off, they are given weight and help to set the tone for the story to come.

This then led to another decision. How to set up the Table of Contents? The story titles and introductions were a no-brainer. But what about the epigraphs? List them as “Epigraph: Story Title” or just use the story title or how about the first line of the epigraph itself? So I asked myself, how do I find quotations when I’m looking for inspiration? Either by subject or author. Since the stories are the subject, I chose to go with the author’s name. It’s my hope that readers will be intrigued by the included names and perhaps find it useful in case they wonder, “Now what was that line from William Shakespeare?”

Now the arrangement was reader friendly. On to making the ebook look good. Make it look worthy of the material.

As anyone who’s produced a Kindle ebook knows, choices in design are limited. Typographical choices are limited, with the standards being Times New Roman (serif) or Arial (sans serif); and it’s best to limit font sizes because the conversion program can get pissy when given too many options. I went with 12 point Times New Roman–serviceable and easy to read. Because I have no control over justification or even how much text a reader opts to show on the screen, I went with consistency over attempts at fancy. I happen to think that narrow indents look better than wide indents, so I set paragraph indents at .3″. I also had to decide how to set off quoted material within the text. My first instinct was to set it off with a wider indent. The danger there is the way the Kindle justifies text and word wraps. In most of the quoted text, the lines are so short neither justification nor word wrapping was a problem, but there were a few long lines. Since I wanted consistency throughout, I went with no indents and italicized text.

When comparing it to a printed book (with kerning and a human hand fiddling with it) it’s not the same. But in an ebook, it works well because it is consistent and even if the reader increases the size of the text, there’s less chance for words to go staggering all over the page. I had tried setting off the quotations further by inserting a line before and after, but felt it set it off too much and made it look disconnected.

That was about as far as I could go with the limited layout options. So that left small details to play with.

Mr. Block used a clipboard and gavel graphic for the book covers. I used it to create the title page and story titles.

I chose AR Julian for the title font because it’s meaty and masculine, but elegant, too, and I thought it complemented the tone of the stories. Notice, too, the “running heads.” One problem I have with Kindle ebooks is I sometimes forget the title of what I’m reading. Unlike some other ereaders, Kindle doesn’t insert true running heads on the pages and there is no way for the producer to insert them. So what I did was insert a faux-running head at the top of the introductions and epigraphs, and at the beginning of each story. Because I’d put the author’s name in the story title graphics, I left-justified the story running head and left off the author name. I thought the slightly different arrangement would help to delineate the story from the introductory material.

I also made a graphic to use for the scene break indicators. I think producers should always use some kind of indicator for scene breaks because there is no way to control the amount of text on the screen and sometimes line breaks can be lost when the reader changes the page. I could have used asterisks or pound signs, but Martin H. Ehrengraf is a bit of a dandy and needed something to complement his elegant clothes and formal manner of speaking.

Notice, too, that at the beginning of each story and scene I removed the indent and bolded the first three words. I have tried faux-drop caps (made by increasing the font size by two points and bolding the letter) but there lies danger. If a reader changes the font size for readability, there is a risk of a hiccup. Bolding three words and not indenting the paragraph is a simple way to set off the text and indicate a new beginning.

I happen to enjoy back matter. I read it all. I’m sure many other readers enjoy it, too. With no concerns for paper or printing costs, there is no reason to skimp on the back matter. In this case, the author included an Afterword, information about himself and a list of titles and links. I included a photo of the author.

The author photo is in color. It looks very good, nice and clear, but I believe a black-and-white photo would have been better. Black-and-white photos, with their lighting suited for gray-scale, look great on the Kindle screen. Something for authors to keep in mind the next time they have an author photo taken.

Overall, I think I achieved my goals. Reader Friendly and Worthy of the Material. The real question is, Does it matter? Plain formatting takes a few hours at most, depending on how clean the source file is–producing this book took me several days. Besides, it’s the stories that matter, right? As long as the stories are good, does the fiddling and tinkering and rearranging and fancy bits make any difference? You wouldn’t serve fine wine in a chipped jelly jar, right? Or serve filet mignon on a paper plate with sauce slopped around willy-nilly and few burned potatoes on the side? Presentation matters in food and it matters in literature. Limitations in Kindle ebook design notwithstanding, with care and thought, the overall reader experience can be enhanced and the stories themselves are well-served. To me it’s well worth the extra time and effort.

Mr. Block tells me he will be making the very first Ehrengraf story, The Ehrengraf Defense, FREE on Amazon for a limited time. That should happen on Thursday. If you can’t wait that long, all eleven Ehrengraf stories are available as singles, or you can find them all in one collection, Ehrengraf For The Defense. Fun to read and it looks great, too.

Scrivener and the Ebook

I heard about the Scrivener writing program last year. I followed a link to Literature and Latte, roamed the site and got interested. What finally hooked me was the promise of a new way to organize a writing project. So in January I purchased the program. (Why so long? Because I’m scared of hardware. Go ahead and laugh, the old man does. Then again he’s a mechanical genius and never worries about the damned things eating him while he sleeps. When he starts fiddling with things, I have to leave the room, freaked out about him pissing off the machines. Because I’m convinced my computer is just waiting for me to do something stupid, it takes me forever to work up the nerve to install new programs. Now you know.)

As a novelist I’m not exactly tidy. My process involves notebooks, Post-it notes, scraps of envelopes, colored pens and pencils, sketch pads, clippings, piles of reference books bristling with place markers and file folders. Rolls of butcher paper, colored Sharpies, white boards and reams of printed manuscript also play a role. Clutter is part of my process. I need to see it, need to have it where I can put my hands on it. That’s what Scrivener does. It digitizes my clutter. Instead of having it scattered on my desk top, it’s on the screen, available with a click of the mouse.

Scrivener is not a word processor. It’s a writing program. It’s not for generating printed documents, it’s for generating files. (You can use it to create printed documents, but using Scrivener to write a letter to your mom would be like using a bulldozer to dig a post hole.) It very easily and quickly generates lots of different types of files, including PDF, RTF, Word, epub and mobi (Kindle) files.

Me, being me, I got ideas. Once I got interested in self-published ebooks, I had to try it for myself. I’m always interested in how things are made. Since I read almost exclusively on my Kindle now, I see a wide range of quality in ebook production. When I see a problem, I try to figure out the source of the problem. If you’re a regular reader, you have seen my obsession with em dashes and spacing issues and ways to exploit the ereader vehicle. To me, the two biggest concerns for any ebook formatter should be: Readability and easy navigation.

There are many ebook formatters who are familiar with programming, coding and HTML. I’m not one of them. I have no idea what goes on behind the screen. Because of the limitations of MS Word and the often interesting glitches that occur when converting Word files, learning ebook production increased my vocabulary (children, cover your ears). I also had problems organizing layouts. Every change made in Word offers the program new chances to screw things up. So I was figuring out HTML and basic programming, but slowly.

Then came Scrivener. It speaks my language and doesn’t require me to memorize codes and commands. With its organizational capabilities, I saw a way to make more than a simple ebook. I could make beautiful ebooks. It took me a few tries to figure out the possibilities until I achieved something that came close to matching my vision. You can see my latest creation here.

I also discovered a few quirks and limitations. BUT, by using Word’s powerhouse Find and Replace feature to take care of spacing issues and oddball punctuation, then stripping the file in a text editor, I can produce a squeaky clean file ready for layout in Scrivener. (And yeah, I know, it’s not particularly efficient, but I can do it very quickly and it makes sense to my peculiar way of thinking.) Scrivener’s special character map is far better than Word’s, too. So while I’m proofreading it is very easy to make a little cheat sheet off to the side for any special characters needed, then a simple search and replace takes care of those. Also, Scrivener’s formatting is basic, without all the desktop publishing features of Word (full of cute little traps that can make a total mess of an ebook). Since ebook formatting (for fiction) is quite basic, too, Scrivener’s simple design is perfect. I created a template so all I have to do is break up the main file, then move sections around to get the layout I want. I can send an RTF or PDF file to the writer for their approval, and if they want things changed around, no problem at all. Minimal risk of screwing up the formatting.

Graphics are a breeze with Scrivener, too. I’m not talking about covers. I’ve tried my hand at making ebook covers, with decidedly mixed results, but that’s a whole other project I’m working on. I’m talking about such things as fancy font chapter heads, scene break indicators and illustrations. Graphics open up a whole new world of possibilities to make an ebook visually appealing. I have only dabbled in graphics thus far, but I can see the potential and have several interesting experiments in mind.

Granted, not every indie author is interested in learning how to format their ebooks. Formatting isn’t difficult at all, but there is a learning curve and a million and one little details to track. If you want to focus on the writing and hire out the production jobs, that’s fine. Formatting isn’t terribly expensive and won’t break any writer’s bank. If, however, you’re a die-hard do-it-yourselfer, but you aren’t adept at programming and coding, Scrivener is an excellent way to go.

If anyone who uses Scrivener has tips and tricks for ebook production, or would like to know exactly how I created Beauty and the Feast, leave a note in the comments. I can write another blog post.