Improved Formatter

Installed as a beta feature on this server as of 2005-06-12.

This is the development page for an improved version of "the Formatter", specifically, the code in ./formatters/wakka.php (as opposed to the AdvancedFormatter page which deals with "advanced" formatting in other ways as well, such as standardized code generation utilities).

Why?

While our current (version 1.1.6.0) Formatter is quite capable, it has some quirks and bugs, doesn't always generate valid XHTML (though it tries hard), and misses a few things that would be nice to have or that would enable things that would be nice to have (such as a page TOCs). The improved version presented here tries to address some of these issues (with more likely to follow).

What?

Here is a short summary of what has changed (details below):

• using single quotes wherever possible making RegExes and generated HTML easier to read;
• better closing of open tags at end of document, including open indents and lists (a long-standing bug!) Now improved
• better handling of nested lists so change of list "type" is actually detected and coded correctly; also produces nicely-formatted HTML code for lists and indents now, especially more readable for nested lists. New!
• escaping single & (not part of an entity) (another long-standing problem);
• ability to nest one type of float within another (so a right float can contain a left float and vice versa)
• handling ids (and making them unique) as provided in embedded code, using the makeId() method;
• creating ids for headings based on content ('afterburner' type formatting so this includes originally embedded code); this code not only uses the makeId() method but also the html_entity_decode() method in PHP versions older than 4.3.

The code presented below is still considered a beta version and as such contains many lines of (commented-out) debug code. These will of course be removed before final release. Any reference to line numbers is (for now) to the new (beta) code since this is a complete drop-in replacement for the original file.

Closing open tags

The current version (Wikka 1.1.6.0) of the Formatter has a bit of code contributed by DotMG to close any left-open tags at the very end of a page. While that can solve some problems with rendering and including pages, the code was incomplete in which open tags were closed. A particular problem was still-open lists and indents which weren't handled at all (see "List parsing bug?" on WikkaBugs). Also, this code would directly echo output instead of returning a string as the rest of the Formatter's main function does.

The new version addresses all of these problems.

Closing of indents and (open) lists was already happening when encountering a newline that doesn't start with a TAB or a ~, so this bit is separated out as a function. Improved now by removing superfluous variables and corresponding parameters.

The section that handles newlines now only needs to call this function:

To close open tags at the end of the page, the new code now calls this function first, and then handles all other open tags, in an order to at least minimize incorrect tag nesting (but see "Not a compete solution!" below):

This is now used like this:

Not a complete solution!

A big problem remains, however: in order to produce valid (X)HTML, open tags cannot just be closed anywhere: there are rules for which elements can contain which other elements. For instance, an inline element (like <em>) can never contain a block element (like a list). So if the inline element is left open (which happens if someone types // to start emphasized text but doesn't close it before starting an indent or list), closing the generated opening <em> tag at the end of the page may prevent display problems in some browsers, but the result is still not valid (X)HTML. This type of problem can only be really addressed with completely different mechanism for a formatter. This should definitely be tackled at some time, but is outside the scope of the current improvements which are designed to work within the current Formatter's mechanism.

Better handling of nested lists and indents

New as of 2005-07-12

There were still some issues with nested lists and indents, in particular when the type of list changed without a "level" change or when changing to a higher level ("outdent"). At the same time, a list or indent right at the start of a page was not detected or handled at all. (Although that is bad style, and a page should start with a heading, it still should be handled correctly by the formatter, of course.) Finally, accented (Umlaut) characters were treated as a list type.

In order to detect a list or indent at the start of the page as well as after a newline (and avoid Umlauts) the detection RegEx for a list or indent is now coded as follows (in the wakka2callback call):

By using (^|\n) as an anchor for matching instead of merely \n the start of the page is also matched.

The actual code for handling a list or indent line was comnpletely rewritten to properly handle change of list types and to produce readable and nicely indented XHTML code. Note that this section now also starts with the (^|\n) anchor:

Since the new code avoids adding an extra <br /> before a list (ul, ol) or indent (div) - these are block-level elements and line breaks should not be used to separate them (they really should be used only within flowing text) - the stylesheet had to be tweaked a little since it actually (implicitly) assumes a line break is there. Change the following in css/wikka.css or your own "skin":

to:

(Since we're dealing with beta code anyway, line numbers refer to the stylesheet as implemented on this server.)

Also, the styling for inline comments (line 66) should be moved so it actually overrides the generic style on line 100 etc. instead of the other way round:

Escaping single ampersands

While there are a few cases where it's actually allowed to use a plain & in HTML, in most cases where an ampersand is not part of an entity reference it needs to be escaped as &amp;. The current (version 1.1.6.0) Formatter escapes the < and > special characters, but not &, so the result may be invalid XHTML.

We need to find the ampersands that are not part of an entity reference. So we first build a RegEx to recognize the part of an entity reference that follows the ampersand that starts it; it can be a named entity, or a decimal or a hex numerical entity; and it can be terminated by a semicolon (;) in most cases, but there are a few cases where it's legal to leave off the terminating semicolon. To make it easier to read, we build the RegEx to express all that from its constituent parts:

So now we can define a 'lone' ampersand as one that is not followed by the expression $entitypat:

This then becomes part of the big expression that's used in the preg_replace_callback() near the end of the file, as the last thing to consider before a newline:

Now we can "escape" all HTML special characters, as we should:

Nesting floats

I happened to find that the code for a left float (<<) would terminate a right float (>>) and vice versa. Which would of course likely leave unclosed tags. It turned out that by solving that it actually became possible to nest unlike floats - one level deep, at least. No great feature, but it could be handy at times.The solution is actually quite simple: there was just a single "trigger" to keep track of start and end of a float; keeping a separate trigger for left and right floats (an not generating newlines) is all that's needed:

Note line 114 where we now use a $trigger_floatr instead of $trigger_floatl: this solves the bug and creates a new micro-feature at the same time.

Now that the improved formatter has been installed as a beta feature, I added a small demo in the SandBox (in case it disappears: look for the edit made on 2005-06-14 21:45:57 in the revisions). --JW

Ids in embedded code

Since in ID must be unique in a page, embedding HTML code, and combining that with generated code, creates a problem. In order to to ensure that the page is valid XHTML, every id attribute must have unique value, regardless where it's coming from.

When generating code that should contain ids, this is simple: just use the makeId() method to generate one, with or without specifying parameters. Still, the result could conflict with id attributes in embedded HTML code so we must handle those as well.

We analyze the whole block of embedded code, run each id through makeId(); if this method detects the id already exists, it will return an amended value with a sequence suffix; if it finds the id value wasn't valid, it will create a new one and return that. The formatter then replaces every id for which a different value was returned:

As long as ids in the embedded code are valid and unique, they remain unchanged because makeId() is called with the 'embed' parameter which tells it not to add an id group prefix.

Still, it is important to remember that ids can only truly be guaranteed to be unique if every bit of code that generates HTML with ids is actually using the makeId() method - and that includes user-contributed extensions.

The "Fatal error: Call to a member function on a non-object" bug referred to on WikkaBugsResolved is also fixed here (line 282).

TODO: There is still one problem to be solved here: when embedded HTML code contains an id, it's entirely possible that it (or a following embedded section) contains a reference to that id. When the makeId() method finds it is necessary to change an id because it conflicts with a pre-existing one, any reference to it should also be updated.
The current code does not (yet) take care of this - it's a fairy complicated problem to solve correctly, but will be tackled soon.

Heading ids

Creating ids for headings is (you guessed it) the first (and necessary) piece of the puzzle to enable generating page TOCs, but other bits will be needed for that as well, such as actually gathering the references to headings (and their levels), and the ability to link to page fragments (something our current core∞ does not support yet). So: we cannot generate TOCs - yet - but we are getting there; the code is also designed to make it possible to extend it to generate TOCs not just for headings, but also for things like images, tables and code blocks.

A method for generating a TOC has not been decided yet (we may even provide alternatives), but one thing we certainly need is ids for headings (see TableofcontentsAction for more background on this); and even if we do not (yet) generate a TOC, being able to link to a page fragment (the obvious next step) will be useful in itself.

Some thought went into the method of generating the ids: Ideally they should be 'recognizable' so creating links to a page fragment with a heading wil be easy, and they should be as 'constant' as possible so a link to a section remains a link to that section, even if that is moved to a different position on the page, or another is inserted before it. This implies that all methods that simply generate a sequential id will not fulfill our requirements. We also don't burden the writer with coming up with ids (or even needing to think about them): they should be able to just concentrate on the content. Instead, we use following approach:

• The actual content of the heading is the basis for the id: this makes it very likely the id wil remain the same even if page sections are re-arranged or new sections inserted.
• A heading may contain images; if the images have an alt text, they are replaced by this (after all, alt text is meant for precisely that: to replace an image where it cannot be shown!); all other tags are simply stripped.
• A heading may also contain entity references; where possible these are translated into ASCII letters (using html_entity_decode()) while all other ones are removed.
• Any character (except whitespace) that is not valid in an id is then removed: the result is a string that consists only of characters valid for an id - but it could now possibly be an empty string
• A valid id can only contain letters, numbers, dashes, underscores and periods, and must start with an (ASCII) letter; this implies that spaces (and whitespace in general) is not allowed. However, the makeId() method tackles this by first transforming all whitespace into underscores.
• The resulting string is examined for uniqueness within the group of "heading" ids; if necessary, a sequence number is added, or a new id hash is generated.

All this is implemented as an "afterburner" type of formatter which is applied after all basic formatting has already taken place and we already have the XHTML output of that process. This ensures that all headings are taken into account, whether they are generated from Wikka markup or from embedded HTML code. The afterburner preg_replace_callback() function is designed to be extended with other types of code fragments we might want to generate ids (and maybe page TOCs...) for.

The 'afterburner' function is defined like this:

This is called (after the primary formatter) as follows:

The result is an id that is almost always derived directly from the heading content, giving a high chance that it will remain constant even if the page content is re-arranged: thus it provides a reliable target for a link.

Keeping track of recursion level

Since the Formatter is now "better" at closing any tags left open at the end of the string it's handling, a new issue is arose turning up with some of the beta code on this server: when the Formatter is being called recursively by an action (Formatter -> Action -> Formatter...) the "second-level" formatter will close all tags that were opened by the "first-level" formatter. When an action using the formatter is embedded in something like a heading or a list element (which in most cases should be entirely valid) the heading or list item (and whatever else is "open") is closed at the end of the action rather than at the point where it should be. While it's not really a good idea for an action (which is interpreted by the Formatter calling the Action() method) to call the Formatter in its turn (and usually redundant), the Formatter should handle this more elegantly and close tags only in its "topmost" instance.

The solution is to keep track of the recursion level and close open tags at the end only at the "outermost" level. The first thing we need is a varable to keep track of the level; we add this at the start of the Wakka class in wikka.php, after the other object variables:

Then in the Formatter, right before the wakka2callback() function is called to do the actual formatting, we increment the variable: