I figured out a better antialiased inking algorithm, which is way faster and I think produces better results:
Updated patch coming soon.
Ooh, that does look good!
If I may ask, how does this one work?
I do think that there are a few areas in which the previous post-process version worked better (the side of the nose, for example), but overall that looks like a very good version, and an improvement over the old supersample version at the least.
Thanks 
Magic! 
Seriously, though, an idea suddenly struck me, leading to a new detection algorithm. Instead of finding min and max values of the quantity of interest in a cardinal-directions stencil (as in 1.8.1 and all the previous improvement attempts), in this one we compare each pixel in the stencil individually to the pixel in the center. Thus, we use more information from the tested pixels.
For each pixel in the stencil, the comparison result is thresholded. The comparison and thresholding for normals is
float3 diff = (cartoon_caux - cartoon_caux0).xyz;
float vote_aux = step(k_cutoff.x, dot(diff,diff));where cartoon_caux contains the normals data from the pixel being tested, and cartoon_caux0 the corresponding data at the center pixel. Since all the normals in the aux texture are normalized to the same magnitude, the test is a nonlinear measure of the difference in direction.
For depth, similarly,
float diff2 = cartoon_cdepth - cartoon_cdepth0;
float vote_depth = step(0.0001, diff2*diff2);As the variable naming suggests, both the normal map and depth detectors vote whether to ink, and these two votes are OR’d together using max() (following the observation that false negatives in the edge detector are more common than false positives).
The resulting yes/no vote is then weighted by the pixel’s euclidean distance from the stencil’s center and summed to a counter. I observed that some form of distance-based weighting of the votes is critical to avoid a blurry look.
Once votes from all the pixels in the stencil are in, the final result is thresholded (to avoid noise, from when only one pixel in the stencil triggers). The range above the threshold is mapped to [0,1], and then processed through a nonlinear remapping function that emphasizes the low end (i.e. makes ink darker than it would be by linear mapping, if only a few pixels trigger).
The upside is that not only does this render reasonably well, but it’s also fast. If the depth detector is off, it’s almost as fast on my machine as the old 1.8.1 inker was. The depth detector does slow it down, but it’s still reasonably fast - unlike the subpixel version.
(With the default stencil size and the depth detector enabled, this needs 24 texture lookups per pixel, whereas the subpixel version needed 72. The number can be reduced to 16 with a slight effect on quality.)
It’s also mathematically correct - there is no need to interpolate normals, as this algorithm samples only at pixel centers.
Ok.
I’m thinking of making the new algorithm the default, but also providing the previous two as optional.
(The supersample version is perhaps not very useful, but it is used internally to render non-antialiased lines (using num_samples = 1) for input to the postprocessor, and with the same settings, also works as a mostly-backward-compatible version for people who want only the bugfixes and some new options, but no antialiasing. Dropping the for loops from that code would not simplify the overall result much )
Updated patch posted to bug tracker.
bugs.launchpad.net/panda3d/+bug/1221546
This is now the final version of this patch for 1.9.0, except for possible changes required after code review.
Sorry that it’s taken me so long to respond–I started an entry for a one-week game development competition on Monday, and have been kept rather busy! However, thank you very much for the explanation of your antialiasing algorithm–it seems rather good. 
One week for developing a game sounds awfully short
Yeah, it seems this algorithm does the job. Looking forward to getting it integrated into 1.9.0.
I have some stuff to take care of in CommonFilters first, so that we can also get ninth’s lens flare in (and some simple filters of my own: CommonFilters - some new filters, and the future).
It is, believe me, it is! XD;