TeleRead: Bring the E-Books Home

First you say that we need a subset of epub because it is too complicated for the average user. Then you describe a hodgepodge of various HTML versions, javascript and “extensible” CSS as an alternative. If you really think that your alternative is easier to deal with, I’ve got a bridge to sell you.

To begin with, epub is based on XHTML 1.1 and CSS 2.1 and the few differences are listed in the spec. The supporting files that are not content are valid XML and are also detailed in the spec.

As for your alternative being an online solution, no thanks. I like to take my ebooks with me and read where there may not be an internet connection.

It is true that the current version of the OPS spec does not include javascript. Depending on your viewpoint, that is either good or bad. There are certainly cases where javascript support would be very useful in an ebook reader. In fact, some type of scripting will probably be added to a future version of OPS.

As for picking on epub for not having scripting, name me one reflowable ebook standard that does include scripting. I can’t think of any. PDF doesn’t count, since it isn’t worth a damn on small screens and isn’t usually reflowable.

Although epub is not perfect (being version 1.0), I think that it is a good beginning and very usable. It is already more capable than older ebook formats like MS Reader, Mobipocket, etc.

One other thing. The valid XHTML and CSS that you create for an epub can be used as-is on a web page. If you want to add javascript or one of the few CSS features that epub doesn’t currently support, that can easily be done. The bulk of the work is already done once you’ve created the epub. Or you can go the other way. Create the web page with XHTML/CSS and use that for the epub. That’s just one reason why using these standards are a good idea, rather than the Wild West days of any-old-flavor of HTML.

What you propose will do nothing to simplify the current ebook situation. It will only add to the e-babel confusion. At least epub stands a chance. It has been adopted by some big publishing houses for internal workflow (and maybe we’ll see it at the consumer level). Two of the most popular ebook reader devices (Sony 505 and Cybook) are due to have native epub reading capability soon.

As a long-time contributor to the OEBPS specification (version 1.0 released in 1999), and the recently released OPS/OPF/OCF specs (which are successors to OEBPS and which underlie EPub), let me address, from my unofficial perspective, points made by both Aaron and Joseph.

First, I’m glad and excited to see a significant number of developers taking an interest in EPub. This is a good sign that we will see, in the future, new EPub authoring, presentation and conversion applications. EPub is an excellent, high-fidelity format for both direct rendering and for user-side conversion to other formats for particular platforms such as very limited resource handheld devices.

Aaron noted a lack of script support as a demerit on OPS. I do understand his reasoning. Let me explain that the reasons we currently have an effective ban on script support in OEBPS/OPS has to do with

• 1) not wanting to overburden publishers and developers at this early stage of EPub implementation, and
• 2) to inhibit potential “proprietization” of the specs (by some “800# gorilla”) via the use of scripts.

We prefer that reading system developers innovate within the XHTML+CSS framework, and if they want some fancy “dancing bears” stuff, to approach the OEB Working Group and request the new feature, then the Working Group will consider how to implement it which benefits everyone in a balanced manner.

Certainly, as EPub support grows, the OEB Working Group will reconsider its position on EPub script support, but it will (and should) take a hard-nose position as to what is gained that cannot be gained by reading system-side innovation plus better CSS support (e.g. CSS3), and of course the impact of scripting on EPub authoring and reading system developers. Yes, it sounds a bit harsh, but it is better to be conservative with the spec in the beginning, and then expand it carefully over time as needs require, rather than just opening it up from the start and then regretting that we had done so later.

I do hope Aaron will contribute requirements to the OEB Working Group so it has something to chew on when it reconvenes to update the current specs. This will include specific things that he’d like to see that scripting would give – that is, not just ask for scripting support, but list what sort of things he’d like enabled (which could be done by scripting.) This gives us a better idea of how to intelligently expand the spec, and whether or not we should bite the bullet and add scripting support.

Joseph, in general I agree with you regarding reading system requirements.

It is important to not go too far in either direction. One can over-specify what a reading system must and must not do, but then this stifles innovation. On the other hand, without any rules, it becomes a free-for-all and EPub authors cannot rely on some sort of presentation uniformity across competing reading systems when they design their EPub Publications.

A balance is thus necessary. We can and should discuss whether the proper balance has been achieved.

And yes, CSS3 will provide several of the “bells and whistles” that scripters would like to enable, at least those that make any sense. For now the “we-want-scripting-in-EPub” supporters need to make a stronger case, detailing exactly what sorts of things scripting will enable that cannot be enabled with CSS3. Such a list will enable the OEB Working Group, when it reconvenes, to decide what to do.

Thanks for clarifying, Aaron.

Actually, I agree with most of what Aaron says. (And yes I do know the difference between “specification”, “recommend”, and “standard”, but in my previous reply I did not want to delve into the intricacies since I was communicating to a wider community who may not understand these nuances.)

My big problem with the current “EPub” is that it is essentially linear in focus – it forces textual content to be mostly linear even when it isn’t. And most textual content is NOT linear in structure/organization. Imagine trying to fit a typical web site into the EPub paradigm. Or how about a snapshot of the Wikipedia?

Because of this strong emphasis on linearizing content, new EPub Reading Systems will focus on linearity – it may lead to EPub not being able to evolve to properly handle non-linear publications (such as representing web sites) because of the already installed base of linear Reading Systems and their vested interests. It’s a real concern of mine.

For example, the OpenReader spec (currently on ice) specified two modes for any publication: “OEB” (content is essentially linear per the OEBPS paradigm), and “WEB” (content should be displayed in web fashion, like a web browser even if discretely paginated on presentation.) This would have forced OpenReader user agents to support both paradigms from the get-go. This should have been done with EPub, but the major players in the working group were advocates of mostly linearity (Adobe and ETI) and I was out-voted. I believe this is a major blunder in the EPub standard.

It is interesting Aaron brought up web browsers. I’ve contemplated writing a blog article describing a strategy to allow EPub Publications to be viewed in web browsers. In fact, it is quite trivial, and I’m amazed no one has (apparently) considered this yet.

First step is to develop an EPub–>WebSite converter, which simply unpacks the EPub (it is, after all, simply a ZIP file), does some very minor link additions to the content documents, converts the NCX to one or a set of table of contents pages, etc., and then saves the results into a folder. A web browser can then view the contents of the folder just like any web site.

The next step is to create a browser plug-in which does the same unpacking but adds some features to make it more seamless for the end-user. There are more advanced things that could be done after this.

The downside to the above is that content documents will be viewed in usual web fashion, typically requiring scrolling to read a chapter or book. (Many people don’t mind this – thousands of books are now online as HTML content and are very much readable.) But still, it’s another way to view EPub Publications which is a Good Thing™. We can imagine plug-ins for IE, Firefox, Opera and Safari, to name the major ones. And, more importantly, imagine now that small handhelds with web browsers will now be able to view the converted Publication since it is simply a “web site”.

In addition, and this is important, web browsers are capable of nicely handling the one non-linear feature included in EPub: auxiliary content. Such auxiliary content can be viewed in separate windows (preferably popup windows), which is the intent of OPS. Here’s where web browsers will be in far advance of Adobe’s DE, for example.

Again, thanks for clarifying several things, Aaron.

Hey Jon -

Just a note on your comment about making epubs readable in web browsers. That’s actually the core of what we developed at BookGlutton. Our import and upload tools accept doc, html, rtf, and txt and produce epub files without the need for plugins. At this time we haven’t turned on the ability to download the epbus, but if our community is interested in that feature we would definitely take it into consideration.

Arron makes very good points concerning epub format.

Agreeing mostly with him, I would like to express my opinion in more general terms: epub will most probably not become a standard for digital publishing because instead of trying to propose a format which is simple, powerful and flexible, IDPF proposes something which is neither. Moreover, epub will most probably never to evolve into simple, powerful and flexible format because IDPF takes patronizing and arrogant stance of “we know best what what is really needed for digital publishing and what is not”.

Scripting is very good example of both failure to provide flexible format for digital publishing and inability to understand the needs of possible adopters of that format. Saying that scripting is needed only for bells and whistles (fancy “dancing bears” stuff) or that instead of scripting some functionality of reading systems must be used, is completely inadequate. Such reasoning would be adequate in early nineties but not now when we have javascript in browsers for a decade. Scripting is *essential* for many digital publishing projects and not understanding it is a major failure of IDPF. Saying that “we will reconsider scripting when adoption of epub grows” is also inadequate, because nobody will wait patiently, but will choose some another platform for their publishing needs, Adobe AIR for example.

It seems to me that all those who think that IDPF goes wrong way should consider forming a new body and formulating specification of a format alternative to epub. Arron has very good ideas about how such format should look like. So, Arron, push your ideas for a new format till a new spec a a new body around them is formed. You will certainly have a following since you are mostly correct and what you propose is simple, powerful, flexible and relatively easy to implement.

Btw, in looking again at Aaron’s BOOK proposal, as best I see his idea is to place all content into a single document, index.html.

One reason OPS allows multiple documents is that it makes it much easier for reading systems to only parse what is needed for the moment, very useful for limited resource hardware. Even Adobe DE for desktops works better when the content is split up (such as into chapters.) Now with a single document approach one could ask authors to split up their content in a document and “wrap” each piece within a specially-classed <div>, but this is not practical for reasons I need not get into.

In addition, in the future, we’d want the spec to be able to represent web site structures, and of course that is multiple files.

The lack of something equivalent to the OPF Package is also troubling. Back in 1999, the first OEBPS proposed was essentially what Aaron proposed (but did not include scripting.) It became clear as the requirements came in from publishers and other stakeholders that there needed to be a “Package” (metadata, identifiers, content organization, etc.)

In addition, for both accessibility and use by the sighted, the recent addition of NCX is revolutionary — we now have a means to have a machine readable table of contents (and similar lists) in every publication, far superior to having the publisher customize something in content itself (thus making it near impossible for accessibility purposes).

When we add all these things together (and many I’ve not even discussed), the idea of a Package is a great one, and I assume this is part of the “unneeded” complexity I hear voiced in this comments area.

Anyway, I could go on, but there are reasons for why EPub is the way it is. Those who believe there is a better way (and I actually do: OpenReader) should strive to understand the requirements that were used, to reevaluate those requirements given what we know today, add new requirements, then see what shakes out for the “next-gen” design. Maybe javascript will be a major part of that new design, maybe it won’t. (As I noted, from the beginning the OEB Working Group was concerned with proprietization of the spec using javascript — we had long discussions on this, a lot of “what if” scenarios involving 800# gorillas.)

Laisvunas, I feel your pain, but I don’t think forming another standards body and a new format is the answer. What I’d like to see is a sort of epub spinoff, another specification from the IDPF, if you will, with slightly different requirements. Instead of BOOK, we could call it epub-lite. The basis for this simplified, consumer-oriented version of .epub would be the same browser-centric building blocks under the IDPF specs. The difference would be in the file structure and in the way a browser deals with it.

Just to be clear, my intention is not to tear down the existing specs or the IDPF. I know developing requirements and specs can take an exceedingly long time, and I don’t think what I propose has to undercut or devalue the work IDPF has already done. I want the tower to topple, but I also understand its necessity as a source of building material.

A nice requirement for the IDPF would have been: “packages must provide a fallback STRUCTURE, in the event that a Reading System cannot natively handle the container.xml and/or content.opf files.” The best fallback, it seems, based on the reach of the web browser market, would be to divert the rendering chain through index.html, since it’s an entry point that all browsers (and in the current spec, all Reading Systems) can understand. Document authors could opt to omit the container and opf files, and the same fallback structure would be used by all Reading Systems. As far as indexing such a structure in a machine-readable way, the spec could name acceptable heading elements from the XHTML rec, such as H1, H2, H3, etc, and build a nested contents structure from those. It could even extend upon XHTML, allowing authors to omit certain headings from the structure (with, say a ‘visibility’ or ‘display’ attribute or CSS property). The beauty of this would be that it still falls under the umbrella of epub, but it allows document authors to stay in familiar territory, and it better follows the principle of XHTML, which is to be extensible, not absolute.

As far as scripting goes, the spec could only allow execution in browser fallback mode. However, even then, allowing it only in the Reading System and not the BOOK would still be fine. @Jon, Re: Scripting. Assuming you don’t want to wrap things in a plugin or a custom build, a browser-based epub Reading System MUST have scripting. There is no way around it if you ever want to get closer to true typographic precision and layout control. You COULD use XSL or CSS only, if they had wide enough and normal enough implementations across user agents and platforms. Even Digital Editions, an 800# gorilla project if ever I saw one, uses a programming language (XSL) to help with layout. And unlike ECMAScript, XSL-FO is not only non-standard but also highly controversial among layout and interface experts, not to mention unheard-of to most browser engines.

If you’re building a browser-based RS, there’s also a server side requirement (though I really wish there didn’t have to be), to handle the container format. While I agree a container format is a requirement and a blessing, using ZIP is very problematic for browsers.

The reason I make this proposition is not out of mere whimsy. This sketchy proposal actually comes from a lot of hard tinkering and coding. I have been building a browser-centric reading system for a year now and as much as possible I’ve followed the specs but I’ve come to the realization that this is a bit of a square peg situation, and it could get worse.

Anyway, if there’s enough interest, I could formalize this more, but my intention is not to work against the IDPF.

Aaron, regarding a container format which web browsers can easily be modified to handle, what would that be?

During the OCF development, we looked at a few container strategies, notably ZIP and MIME-based.

Obviously the issues of server vs. client side play a role.

Thanks. Yes, the OCF Working Group looked at a variety of ways to wrap an OPS Publication into a single file for distribution purposes. Lee Passey was also involved in some of the working group development and hopefully he will weigh in with his thoughts.

We settled on a modification of what the Open Office group was using, which has its roots in JAR (based on classic ZIP).

Bill Janssen continues to advocate a MIME container for this purpose, and if he is reading this, I hope he weighs in as well.

Laisvunas, you have a really good point there. WHATWG is an excellent example of how one can spur on another (though remember they released the public draft on HTML 5 together).

The system I’m referring to is alive and well at bookglutton.com. It features an AJAX reader and Package Creation tool. The package tool is currently part of the upload feature which enables people to convert .doc, .rtf, and html documents to epub packages that can be viewed in the Reader. Once we have more epubs out there, direct epub upload will also be an option. We may also eventually enable epub download. Right now, we’re having some doubts about the value of that.

I agree with your thoughts on scripting. I think anyone who develops for the browser would share those thoughts too. I think one problem is that not many e-book reading system developers come from a browser-centric background.

Joseph, I never stated that I saw no value in epub.

As for content, all of our content is free and no account is required to read it. For those who like to read offline, I highly recommend supporting your local public library or bookstore.

To quote from your original post:

“but we also need legions of independent developers building APIs and authoring tools and Reading Systems and open-sourcing all of them.”

I agree with you 100% on this. How about sharing some of those tools that you have developed?

Aaron, have you looked at Openberg Lector? It is a plugin that allows you to read epub (and a few other formats) from within Firefox. I have tried it and although it is usable, it needs more work. Perhaps you can either contribute to that, or use some of their ideas as a basis for your own reader.

I took a look at your site and the presentation is quite nice. From my perspective, it is a shame that I can’t download these nicely formatted ebooks to read away from my PC. I’ll have to find them elsewhere.

Although OPS does not “bless” any particular audio/video format, they may certainly be used in OPS Publications. Fallbacks do need to be provided, however.

About web browsers, it certainly is possible to build plug-ins for web browsers that can render EPub. I discussed this in a prior comment.

Jon, while we are all discussing epub, what about dictionary support in the next version of the spec? As I commented on in another blog entry here, I think a standard dictionary format and lookup method really needs to be added to epub (and soon).

Some people don’t seem to care about having an integrated dictionary for e-reading and some can’t be without one. I’m more in the later camp.

The dictionary formats used by Microsoft Reader and Mobipocket Reader are both well documented. The easiest thing to do would be to adopt one of these, thus gaining an instant selection of commercial dictionaries for purchase. If that option is not possible or desireable, then certainly some similarly structured XML format can be invented for epub reader use. Or, even one of the existing XML dictionary formats that the Open Source community has developed (at least two that I know of) can be used.

The mythical scenario you “suppose” about would render differently in each of today’s browsers as well (as I previously pointed out). If you want such typographic control of your content, PDF is your medium, not the web. Your arguments are specious. I’m done.

>Can you give us some specific examples of what you consider to be adequate and inadequate readers?

FBReader is inadequate since it does not support HTML tables. Only Openberg is adequate for rendering pages since it is based on Firefox as its plugin.

>you’re objecting to the fact that different rendering engines produce different results, well, that’s just a fact of life. Web developers have been dealing with this problem for years, and it’s not likely to get better any time soon.

On the web this problem is getting better with a release of every new version of any major browser. In epub space this proble is getting worse with release of any new reader.

>The current epub spec, despite its shortcomings (which are likely to be addressed in subsequent versions), strikes me as a perfectly reasonable first pass at defining a file format designed primarily to support books as we know them today — an ordered, linear collection of pages — containing text and images; in electronic format. It does not attempt, nor should it, to revolutionize the concept of a book.

Epub format was proposed and in some circles accepted as a kind of standard for electronic publishing. I dispute that epub deserves such status. To propose as a standard some format which is designed to support books as we know them today is wrong. Nobody knows what electronic publishing will evolve into and how electronic publications will look like. In this unsettled situation those who propose some kind of standard should leave as much as possible space for experimenting, should strive to provide maximum flexibility.

IDPF does not go in this direction. They propose utterly restrictive format and say that it is right to adopt it now. I dispute this. It is wrong to adopt epub now for many reasons. First, its unflexibility and restrictiveness, second, appalling state of reader software. Only Openberg is based on browser engine which rendering power can be believed into (but it is immature and it is doubtful even if its developers will release a version compatible with Firefox 3 when the latter ships); the other readers are based on something nobody knows what. The common denominator of what can be displayed reliably on epub readers is very low (no html tables, no css positioning). On the web situation is quite different: although there are differences between browsers, the common denominator of what can be displayed reliably in major browsers is quite high.