H.264 video in Firefox for Android

Firefox for Android
has expanded its HTML5 video capabilities to include H.264 video playback. Web developers have been using Adobe Flash to play H.264 video on Firefox for Android, but Adobe no longer supports Flash for Android. Mozilla needed a new solution, so Firefox now uses Android’s “Stagefright” library to access hardware video decoders. The challenges posed by H.264 patents and royalties have been documented elsewhere.

Supported devices

Firefox currently supports H.264 playback on any device running Android 4.1 (Jelly Bean) and any Samsung device running Android 4.0 (Ice Cream Sandwich). We have temporarily blocked non-Samsung devices running Ice Cream Sandwich until we can fix or workaround some bugs. Support for Gingerbread and Honeycomb devices is planned for a later release (Bug 787228).

To test whether Firefox supports H.264 on your device, try playing this “Big Buck Bunny” video.

Testing H.264

If your device is not supported yet, you can manually enable H.264 for testing. Enter about:config in Firefox for Android’s address bar, then search for “stagefright”. Toggle the “stagefright.force-enabled” preference to true. H.264 should work on most Ice Cream Sandwich devices, but Gingerbread and Honeycomb devices will probably crash.

If Firefox does not recognize your hardware decoder, it will use a safer (but slower) software decoder. Daring users can manually enable hardware decoding. Enter about:config as described above and search for “stagefright”. To force hardware video decoding, change the “media.stagefright.omxcodec.flags” preference to 16. The default value is 0, which will try the hardware decoder and fall back to the software decoder if there are problems (Bug 797225). The most likely problems you will encounter are videos with green lines or crashes.

Giving feedback/reporting bugs

If you find any video bugs, please file a bug report here so we can fix it! Please include your device model, Android OS version, the URL of the video, and any about:config preferences you have changed. Log files collected from aLogcat or adb logcat are also very helpful.

(Cross-posted on the Mozilla Hacks blog: H.264 video in Firefox for Android.)

60 second tour of B2G’s Virtual Keyboard

This post is a whirlwind tour of Boot To Gecko‘s virtual keyboard implementation. I take no credit for the design. These are just my notes from a recent expedition into Boot To Gecko (B2G).

B2G has a microkernel-like architecture. B2G apps are written in HTML+CSS+JS, each app running in an isolated “content” process. They communicate with a centralized Gecko process that is trusted to interface with hardware, such as drawing on screen or dispatching events to app processes.

B2G’s virtual keyboard consists of two halves: an unprivileged keyboard app and a privileged keyboard component in the Gecko process.

The keyboard app is a regular app and runs in its own content process. The keyboard app draws keys on screen and converts touch events into key events. In the future, users might be able to install third-party keyboard apps. The important files of the keyboard app are MozKeyboard.js, keyboard.js, and latin.js.

The Gecko process’ keyboard component routes key events from the keyboard app to the other apps. The important files of Gecko’s keyboard component are Keyboard.jsm and forms.js.

Take a deep breath before continuing on to the example. :)

Say a user is running the Contacts app and would like to add a new contact. On the “Add Contact” screen, the user taps the “Name” text field. Gecko’s forms.js receives a focus event for the text field and sends a “Forms:Input” message. Gecko’s Keyboard.jsm component receives the “Forms:Input” message and sends a “Keyboard:FocusChange” message to the keyboard app’s MozKeyboard.js. Control now passes from the Gecko process to the keyboard app process.

In the keyboard app process, MozKeyboard.js receives the “Keyboard:FocusChange” message and calls keyboard.js’ onfocuschange() callback, which calls showKeyboard() to draw a keyboard on screen and load an input method for the user’s language. The default input method is latin.js.

keyboard.js receives touch events and sends corresponding key codes to latin.js. The input method analyzes the key codes, then calls navigator.mozKeyboard.sendKey() to return key events to Gecko’s Keyboard.jsm. The input method may synthesize additional key events to implement advanced editing features like capitalizing characters at the beginning of sentences, inserting spaces after punctuation, or correcting misspelled words.

For example, latin.js spawns a SuggestionsWorker thread and sends a “predict” message with the key codes that the user has typed since the last word break. The SuggestionsWorker searches a dictionary for possible matches and asynchronously returns a “predictions” message containing (up to) three predicted words for keyboard.js to display on screen.

Keyboard.jsm receives the key event from the keyboard app and sends a “Forms:Input:Value” message to forms.js. forms.js dispatches a DOM input event to the Contact app’s “Name” text field.

Then the user presses a second key and the whole process repeats.

Comparison of Android keyboard apps’ popularity

To help prioritize my work on Firefox for Android’s keyboard and IME bugs, I created this list of popular Android keyboard apps. The Google Play Store does not divulge much information about other company’s apps, but it does reveal rough upper and lower bounds on the number of “Installs” over the last 30 days. For comparison, I have included numbers for popular Android browsers: Chrome, Dolphin, Firefox, and Opera.

I also compiled the number of 4 and 5 star ratings for these apps. I assume that people who like an app enough to write a review and give it 4 or 5 stars are likely to remain active users.

btw, this list does not include the Swype keyboard because it’s not available in the Play Store. Swype is only available bundled on a phone or from the company’s beta program.

Open questions:

  • Why is the GO Keyboard so popular? It supports many languages and themes, but I would imagine that a keyboard designed for the nuances of a particular language would be more popular.
  • Why does the Dolphin browser have so few beta users compared to Firefox and Opera? About 10% of Firefox and Opera users are using beta versions, but only 1% of Dolphin users.
  • Is there a fair way to aggregate numbers for apps that have multiple versions? Many apps have free and paid or stable and beta versions. These populations likely overlap. For example, many users are likely to have installed the trial version of an app before paying for it. Beta users may keep the stable version installed in case they are blocked by a beta bug.
Keyboard Installs (min) Installs (max) 4+5 Star Ratings
* Opera browser 10 M 50 M 219 K
* Opera Next browser 1 M 5 M 34 K
* Chrome browser 10 M 50 M 89 K
* Dolphin browser 10 M 50 M 793 K
* Dolphin Beta browser .1 M .5 M 8 K
* Firefox browser 10 M 50 M 95 K
* Firefox Beta browser 1 M 5 M 19 K
GO Keyboard 10 M 50 M 160 K
Google Korean IME 5 M 10 M 13 K
Smart Keyboard Trial 5 M 10 M 12 K
Smart Keyboard Pro .1 M .5 M 14 K
SlideIT Keyboard Trial 5 M 10 M 21 K
SlideIT Keyboard .5 M 1 M 13 K
A.I.type Keyboard Free 1 M 5 M 3 K
Google Japanese Input 1 M 5 M 6 K
Google Pinyin IME 1 M 5 M 23 K
Keyboard from Android 2.3 1 M 5 M 17 K
MultiLing Keyboard 1 M 5 M 11 K
Simeji Japanese Keyboard 1 M 5 M 15 K
SwiftKey 3 Keyboard 1 M 5 M 77 K
SwiftKey 3 Keyboard Free 1 M 5 M 23 K
TouchPal Keyboard 1 M 5 M 22 K
Hacker's Keyboard .5 M 1 M 6 K
Perfect Keyboard .5 M 1 M 6 K
Siine Keyboard .5 M 1 M 3 K
OpenWnn Plus .1 M .5 M 1 K
Thumb Keyboard .1 M .5 M 5 K

Adding custom search engines to Firefox for Android

One of my favorite “underdocumented” features of Firefox for Android is the ability to add custom search engines to your “Awesome Screen“. Almost any website with a search form can become a search engine. Just long-press in the search form’s text box and Firefox will display a popup menu with an “Add Search Engine” command:

For example, BoardGameGeek is a popular gaming website with reviews of many, many games. I added BoardGameGeek’s search form to my Awesome Screen so I can quickly look up reviews when I’m shopping at my friendly local game store:

Firefox 18 for Android adds DOM_KEY_LOCATION_JOYSTICK support for game controllers

Firefox contributor Christian Vielma recently fixed Android bug 756504 for Firefox 18. Christian implemented the feature; I was just the mentor. :)

DOM KeyboardEvents include a location attribute which describes the source of the key event, such as DOM_KEY_LOCATION_STANDARD or DOM_KEY_LOCATION_NUMPAD. With Christian’s fix, Android game controllers now accurately report DOM_KEY_LOCATION_JOYSTICK for controller buttons. This will be useful for HTML5 games that would like to support game controllers (like the Sony Ericsson Xperia Play). Firefox 18 for Android is the first Firefox platform to implement DOM_KEY_LOCATION_JOYSTICK.

Firefox can determine whether a key event originated from a keyboard or a game controller using Android’s KeyEvent.isGamepadButton() API. Unfortunately, this API was added in Honeycomb and, unlike Google Chrome for Android, Firefox supports Android versions all the way back to Froyo. Christian wrote an isGamepadButton() “polyfill” method for use on Froyo and Gingerbread.

Also, KeyEvent.isGamepadButton() curiously reports false for some D-pad events such as KEYCODE_DPAD_CENTER. I don’t know whether this is an oversight or Google does not consider the D-pad to be a “button”. Regardless, Christian remapped these D-pad events to DOM_KEY_LOCATION_JOYSTICK to avoid passing on this surprise to Android web developers.

Thanks for your help, Christian!

Firefox 17 fixes Sony Ericsson Xperia Pro’s hardware keyboards

Sony Ericsson Xperia Pro and Xperia Mini Pro users, you have patiently endured with hardware keyboards that could not enter numbers or non-English characters. Good news! I just fixed these bugs (bug 772252 and bug 766317) in Aurora 17. I will uplift these fixes for next week’s refresh of Firefox Beta 16 in the Google Play store.

The problem was that Firefox didn’t remember to “lock” the ALT shift state after the ALT key was released. The workaround was to enter numbers and non-English characters using the virtual keyboard or to hold down ALT while pressing the other key.

Bug 254139: 2004–2012

I just fixed bug 254139, which was originally opened in 2004!

By day, I work on Mozilla’s Firefox for Android team, but after hours I also hack on the Firefox desktop browser.

For eight years, Firefox has saved web pages using the original filename, typically something unhelpful like index.html. With today’s Nightly build, Firefox adopts the decade-old precedent set by other browsers and saves web pages using the human-friendly <title> tag!

For such a small change, this was a surprisingly controversial bug. Some people claimed they switched browsers because of this bug.

Firefox Text Input using Android IME

I’m a developer on Mozilla’s Firefox for Android team. One of my areas of focus is text input. Android input methods include virtual (on-screen) keyboards, hardware keyboards, hand-writing recognizers, and speech-to-text input. Any user control that enables text to be entered is an “Input Method Editor” (IME).

I gave a lightning talk about Fennec (Firefox for Android) and Android IME at a Mozilla work week in Toronto 2012. You can view the slides from my talk here: “Fennec Text Input using Android IME”

IME is a system abstraction that decouples text input from text processing. In theory, this separation allows users to combine input methods and applications that had not been previously tested together. But in practice, all software has bugs.

Fennec sits at the exciting intersection of bugs in Google’s Android framework, third-party developer’s virtual keyboard, and Gecko itself. If you find any bugs in Firefox for Android’s text input, please file a bug report with Bugzilla.

Special thanks to Joone Hur. I linked to Joone’s helpful chart describing WebKit key events (and bugs :) to illustrate the sequence of DOM key events fired during text input: “IME composition events are handled inconsistently in WebKit”.