Improvements in accessibility for Mac OS X in Firefox 41

During a big Mozilla event in June in beautiful Whistler, British Columbia, Canada, a few team mates, contributors and I had the chance to work on some improvements to what Firefox exposes to VoiceOver on the Mac OS X operating system. These improvements will be in Firefox version 41, currently in beta, which will be out in September.

Tables

Semantics for HTML tables have not been exposed to VoiceOver at all so far. This is changing in Firefox 41. Modeled after what Webkit exposes, table structures are now exposed including headers, row and column counts, and VoiceOver navigation among cells is also possible. Thanks to Frédéric for doing the heavy-lifting on this one! 🙂

Various more element roles

In a cumulative patch, I rolled out improvements and additions to what roles, subroles, and roleDescriptions are exposed for various HTML and WAI-ARIA roles. Some of them were already there, but wrong, and others hadn’t been present at all yet. All of these are modeled after what Webkit exposes. And since none of this is properly documented anywhere, the only way to get at this information is to read Webkit’s source code. 🙂

You can expect to see better exposure for things like alert, alertdialog, ARIA grid-related roles, and other elements and widgets that weren’t quite right yet.

MathML

Frédéric did a lot of work to properly expose MathML elements in an accessible form for OS X in this release. His over-all MathML project for Gecko encompasses more than just OS X. He documents it on his blog in part 1 and part 2 here. This project is still on-going.

In the case of OS X support, we should be close to, if not identical, to what Safari exposes to VoiceOver. So hopefully, reading MathML in Firefox with VoiceOver gives the same information as VoiceOver would give in Safari. Note that we have no control over what VoiceOver actually speaks, we can only control what we expose.

If you’re more interested in the progress on the MathML support, I suggest you follow Frédéric’s blog as he progresses through his project with us.

So, are we done yet?

Unfortunately not. Especially one remaining big bug is that we still don’t handle Apple’s rather complicated text interface right. The consequence is that typing into fields works, but re-reading what you typed doesn’t. We do have some text-related exposure, but that is either wrong, outdated, or doesn’t work for some other reason. Unfortunately, we’re still tight on resources. So if anyone wants to step up and contribute a patch for this, she or he is more than welcome to do so! Here’s the bug!

But there are other things that still prevent me (and probably most others) from using Firefox for Mac on a daily basis with VoiceOver. One is a really annoying one. Even though we did have a patch for it, the event we’re firing still doesn’t tell VoiceOver that a page finished loading.

But there’s more. We still are slower than I’d like us to be, especially on busy pages. And we don’t expose any WAI-ARIA live region stuff yet, including in alerts (which at least now have correct roles and subroles). So if you have free time to contribute and would like help out on any of these, please get in touch, and we can take it from there!

In conclusion

The stuff that made it into Firefox 41 is a big step forward. We made that room to improve just that bit smaller! Yay! There’s still a lot of it left, though. But at least some progress was made. If you feel brave and want to test this in Firefox 41 and above when it comes out as a release, feedback is always welcome. I cannot promise when we can actually act on it, but we’ll at least get it recorded and on our radar. So keep the feedback coming, as always!

Quickly check your website for common accessibility problems with tenon.io

Tenon.io is a new tool to test web sites against some of the Web Content Accessibility Guidelines criteria. While this does not guarantee the usability of a web site, it gives you an idea of where you may have some problems. Due to its API, it can be integrated into workflows for test automation and other building steps for web projects.

However, sometimes you’ll just quickly want to check your web site and get an overview if something you did has the desired effect.

The Tenon team released a first version of a Chrome extension in December. But because there was no equivalent for Firefox, my ambition was piqued, and I set out to build my first ever Firefox extension.

And guess what? It does even a bit more than the Chrome one! In addition to a tool bar button, it gives Firefox users a context menu item for every page type so keyboard users and those using screen readers have equal access to the functionality. The extension grabs the URL of the currently open tab and submits that to Tenon. It opens a new tab where the Tenon page will display the results.

For the technically interested: I used the Node.js implementation of the Firefox Add-On SDK, called JPM, to build the extension. I was heavily inspired by this blog post published in December about building Firefox extensions the painless way. As I moved along, I wanted to try out io.js, but ran into issues in two modules, so while working on the extension, I contributed bug reports to both JPM and jszip. Did I ever mention that I love working in open source? 😉

So without further due: Here’s the Firefox extension! And if you like it, a positive review is certainly appreciated!

Oh, and if you’re into WordPres development or have often-changing content on your WordPress site, I highly recommend you check out Access Monitor, a WP plugin that integrates with tenon.io, written by Mr. Joe “WordPress Accessibility” Dolson!

Have fun!

Accessibility in Firefox for Android: Some more technical background, Part II

A long while back, I wrote a post explaining some of the more technical details of the implementation of the accessibility in Firefox for Android. If you want to read the whole post, feel free to do so and then come back here, but for those of you who don’t, here is a short recap of the most important points:

  1. What made the accessibility possible at all in the first place was the fact that Firefox for Android started to have a native Android UI instead of a custom XUL one.
  2. The only thing that needed to be made accessible was the custom web view we’re using, all the rest of the browser UI gained accessibility from using native Android widgets.
  3. The switch to a native UI also gave us the possibility to talk directly to TalkBack and other assistive technology apps.
  4. At the core is the well-known accessibility API also used on the desktop, written in C++. On top of that sits a JavaScript layer, code-named AccessFu, which pulls information from that layer and generates TalkBack events to make everything speak. It also receives the keyboard commands from the Android side, and as we’ll see below, has been substantially extended to also include touch gestures.
  5. There is now also an extended layer of accessibility code on the native Android layer, which I’ll come to below.

Making Explore By Touch work

After that last post, we had to make Explore By Touch work, for both Ice Cream Sandwich and, as it was released shortly after, the all-new Jelly Bean and beyond accessibility enhancements. For that, the JavaScript layer receives touch events from the Android side of things and queries the core engine for the element at the touch point. Also gestures like tapping, swiping, double-tapping and two-finger scrolling are received and processed accordingly.

For Jelly Bean and beyond, we had to do a special trick to make left and right swipes work. Because we’re implementing everything ourselves, we have to fake previous and next accessible elements, making TalkBack and others believe they have true native accessible elements to the left and right of the current element. TalkBack, in effect, only knows about three accessibles on the page. The currently spoken one is always the one in the middle of these three. When the user swipes right to go to the next element, the element to the right becomes the new middle one, gets spoken, and at the same time, the previous middle one becomes the one to the left, and the next regular page element is fetched and becomes the new right element. This particular logic sits in our Android accessibility code and queries the JavaScript portion for the relevant accessibles.

The fact that we have so much control over this stuff, in fact we have to do it this way or it wouldn’t work, allowed us to port the swiping back to Ice Cream Sandwich AKA Android 4.0, which doesn’t natively support that, and even Gingerbread AKA Android 2.3, which doesn’t have Explore By Touch support at all. But in the Firefox web views, it works! Including swiping and double-tapping, two-finger scrolling etc. Unfortunately, there was no way to make the rest of the browser UI accessible by touch on Gingerbread, too, so you’ll still have touse the keyboard for that.

On Jelly Bean and above, we are restricted a bit by what gestures Android supports. In effect, we cannot change the swiping, dluble-tapping, exploring, or two-finger scrolling behavior. We also cannot change what happens when the user swipes up and down. In those instances, Android expects a certain behavior, and we have to provide it. This is why, despite popular request, we currently cannot change the 3 finger swipes left and right to something more comfortable to execute quick navigation. Single-finger swipes left and right are strictly reserved for single object navigation. We’d love it to be different, but we’re bound in this case.

BrailleBack support

Some of the above techniques were used, in a slightly different fashion, to also implement BrailleBack support. As for TalkBack and friends, we have to implement everything ourselves. You have to implement two protocols: com.googlecode.eyes-free.selfbraille.SelfBrailleClient and com.googlecode.eyes-free.selfbraille.WriteData. This isn’t documented anywhere. Our summer intern Max Li did a terrific job dissecting the BrailleBack code and grabbing the relevant pieces from the GoogleCode project, and the result can be seen in Firefox for Android 25 and later. Max also added separate braille utterances, so the output isn’t as verbose as for speech, and follows better logic for braille readers. A few weeks ago, a review of using Android with braille was posted on Chris Hofstader’s blog by a deaf-blind user highlighting how well he could work with Firefox for Android in large part because of its excellent braille support. To reiterate: max was a summer intern at Mozilla last year. He is sighted and had never been in contact with braille before this as far as I know. He implemented this all by himself, occasionally asking me for feedback only. Imagine that, and then getting this review! I’m proud of you, Max!

And Max didn’t stop there: In Firefox 29 and above, an improvements to the way unordered and numbered lists are being presented in braille.

Much of all of this is good for Firefox OS, too

Because of the layered nature of our code, much of what has been implemented for Firefox for Android can be re-used in Firefox OS as well. The differences are mainly in what comes on top of the JavaScript/AccessFu layer. Talking to a synth directly instead of generating TalkBack events, of course using the new WebSpeech API, and in the future we’ll also “only” need to implement BrlTTY or something similar support and connectivity for braille displays. The abstraction allows us to then put the utterances to good use there as well. The main problem we’re having with Firefox OS right now is the actual UI written in HTML, JS, and CSS, code-named Gaia. Getting all the screens right so you cannot swipe to where you’re not supposed to at any given moment, making everything react to the proper activation events etc., and teaching the Gaia team a lot about accessibility along the way are the major tasks we’re working on for that right now. But thanks to the layering of the accessibility APIs and implementations, the transition from Firefox for Android was, though not trivial, not the biggest of our problems on Firefox OS. 🙂

In summary

The Android API documentation was not much help with all of this. As mentioned, the portion about SelfBrailleClient and friends wasn’t documented anywhere at all, at least I didn’t find anything but source-code references, among them that of Firefox for Android, in a Google search. But also the exact expectations of TalkBack aren’t retrievable just by studying the docs. Eitan had to dive into TalkBack’s source code to actually understand what it was expecting of us to deliver to make it all work.

Here’s to hoping that Google, despite its quest to close-source Android more and more, will keep BrailleBack and TalkBack open sourced so we can continue to read its source code in future updates to keep providing the good support our users have come to expect from us.