All posts by Chris Peterson

About Chris Peterson

I'm a software developer on Mozilla's Firefox for Android team.

Mozilla Firefox/Platform Engineering Update 40.1

This engineering update is also available on the Platform/2015-04-24 wiki.

Firefox Release Schedule

Only two weeks until the next Firefox release: 2015-05-11!

Layout (David Baron)

Off-main-thread animations now enabled on all platforms for Nightly and Aurora (bug 980770). This means many animations of the transform and opacity properties will now run on the compositor thread, with fewer updates on the main thread.

Media

We shipped some MSE fixes in the Firefox 37.0.2 chemspill, so YouTube will re-enable HTML5 video for Firefox 37.0.2 on Windows next week.

Performance (Vladan Djeric)

Aaron Klotz and Ben Turner paid a visit to Adobe’s Flash team at their SF offices. They were able to provide fixes for several Flash player issues! An example of their work is in bug 1133351. This particular patch also landed in our codebase.

Many Telemetry histograms automatically expired in Nightly 40. Check Histograms.json and bug 1156565 to see if your histogram needs to be updated.

Telemetry unification work is continuing, Phase 3 is tracked in bug 1120356. Enabling unified Telemetry (i.e. FHR + Telemetry) might get pushed back to Firefox 40.

Among the latest Telemetry changes, Telemetry will submit data more aggressively to make analyses more reliable (akin to FHR) and to reduce reporting latency. Additionally, unified Telemetry now archives Telemetry data on the client for 6 months — this archive will be used to power the Firefox Self-Support feature.

Yoric added a “Task Manager” feature to Nightly 40 (bug 674779) to identify tabs and add-ons that are consuming the main thread’s time and slowing down Firefox. Specifically, it reports the time spent by the main thread inside different JS compartments. You can see a very rough UI for this feature by opening about:performance in Nightly. For overhead reasons, the feature currently does not measure time spent in compartments belonging to Firefox’s own chrome JS code.

The Performance team, alongside Platform teams, will be studying page scrolling & page navigation performance on desktop & Android in Q2: https://wiki.mozilla.org/Firefox/Content_Performance_Program

The Places database work done by the “Forget” toolbar button, the “Forget Site” history-menu feature, and the “Clear recent history” feature has been moved off the main thread (bug 1076775). Less jank!

If you notice any plugin issues (crashes, videos not loading, etc), please file a bug and mark it as blocking the async plugin initialization feature (bug 1116806).

Mozilla Firefox/Platform Engineering Update 39.3

This engineering update is also available on the Platform/2015-04-14 wiki.

Firefox Release Schedule (lmandel)

  • We shipped a Firefox 37.0.1 desktop and mobile chemspills last week.
  • Firefox 38 Beta 1 shipped last week.
  • Updates have been re-enabled for Aurora.
  • 5 weeks until the next merge date (2015-05-11).

Upcoming Outages (hwine)

Tree Closing Window on Saturday April 18 for at least 4-5 hours of work (time TBD). Tracking bug 1151645.

MemShrink (njn)

Brian Hackett took advantage of the removal of object parents to reduce the size of base shapes (bug 1143256). The JS engine creates many of these to track the characteristics of JS objects.

Nick Fitzgerald greatly reduced the amount of memory used for saving JS exception stacks (bug 1038238). This reduced the memory of a test case compiled from Dart to JS from over 1GB to around 170MB (bug 1125259). For implementation details, see Nick’s blog post “A Compact Representation of Captured Stack Frames for SpiderMonkey”.

Seth Fowler fixed a bad leak regression (bug 1150127) where closing a page while it was in the process of loading could leak the page and cause long browser pauses. Seth also fixed a pair of issues that caused very large memory usage (more than 1GB) when scrolling in the downloads list (bug 1148682 and bug 1148684).

Bill McCloskey fixed an issue where a docshell was being held alive for too long on e10s (bug 1137933). This was one of the remaining blockers to enabling e10s Mochitest browser-chrome tests, which will give us more test coverage for leaks.

Media (cpeterson)

We shipped MSE (Media Source Extensions) for YouTube in Firefox 37 on Windows Vista+. Unfortunately, we discovered multiple unrelated issues in the release channel that all manifest themselves as “video is black”:

YouTube helped us quickly revert all Firefox users from HTML5 video to Flash. Once the problems were identified, YouTube was able to switch OS X and Windows Vista+ pre-channels back to HTML5 video. We plan to ship MSE for YouTube on Windows Vista+ in Firefox 38 (or 37 if hotfix bug 1152630 is released soon) and on OS X in Firefox 38 (or possibly slip to 39).

Mozilla Firefox/Platform Engineering Update 39.2

This engineering update is also available on the Platform/2015-03-25 wiki.

Firefox Release Schedule (lmandel)

We shipped two chemspills over the weekend: Firefox 36.0.3 and 36.0.4 (plus Firefox ESR 31.5.2 and 31.5.3)

Firefox 37 is scheduled to be released on March 31. That’s next week! For Firefox 37 Beta/Release, please report any keyboard related issues on OS X such as the keyboard stopping to work and requiring a browser restart.

MemShrink (njn)

jemalloc3 has been disabled again on trunk due to Windows crashes. The heap allocator is a fundamental thing and changing it in a complex program like a browser is hard.

Thanks to Jim Blandy, jemalloc (the old version currently used in Firefox releases, not jemalloc3) is now used as the allocator for the JavaScript shell, instead of the system allocator (bug 1134039). This makes the shell configuration closer to the browser configuration, which will make arewefastyet.com results more representative. But we don’t yet have data on whether those results have changed significantly due to some OS X problems (bug 1146267). It’s worth repeating: changing the heap allocator is hard.

about:memory now has a “resident-peak” measurement (bug 1145007) on Unix (Linux, OSX, *BSD), which measures peak physical memory usage for the process. It may be useful for detecting short-lived spikes in memory usage.

The new Reader View feature was causing high memory usage on desktop Nightly, but has been fixed in bug 1139678.

Media

In Firefox 37, the Media team is shipping the MSE API (Media Source Extensions, bug 778617) to improve YouTube’s HTML5 video playback on Windows. YouTube will now default to HTML5 video instead of Flash on Windows. MSE for OS X is being tested in Aurora 38.

Shumway

The Shumway team continues to focus on improving startup performance and Flash ad rendering.

Mozilla Firefox/Platform Engineering Update 39.1

This engineering update is also available on the Platform/2015-03-17 wiki.

Firefox Release Schedule

Only two weeks and two beta builds until Firefox Beta 37 is released. The Tracking Firefox dashboard shows tracked bugs for the Aurora and Beta releases.

OrangeFactor

RyanVM sends a major shout out to Tim Taubert for being on an orange fixing tear last week.

MemShrink

Mike Hommey enabled jemalloc3 (bug 762449) on Nightly. It’s configured to not ride the trains for now. There are a number of perf regressions and crashes that Mike is diligently working through.

Media

The media team is close to shipping Media Source Extensions (MSE bug 1083588) for YouTube. MSE improves HTML5 video performance as YouTube transitions its default video player from Flash to HTML5 video. The team plans to ship MSE on Windows in Firefox 37, OS X in Firefox 38, and then Linux. Be on the look out for any YouTube bugs, such as videos that won’t load or get stuck rebuffering.

Performance

Telemetry and FHR measurements were unified into a single system on Nightly 39 (bug 1069869). The old FHR will still be around for a couple of releases. The unified Telemetry client-side is still being stabilized, so Telemetry from Nightly 39 isn’t reliable yet. Be very careful when interpreting Nightly 39 data!

The unified Telemetry pings are sent to a new Telemetry backend (DataPipeline) which will support streaming analyses. The Telemetry wiki is now up to date with information about Telemetry client-side & server-side workings. We added a Telemetry errata page to help with analyzing Telemetry data.

Many Telemetry histograms will automatically expire in Firefox 40. Check Histograms.json to see if the expiry version of your telemetry probes should updated.

Test results comparing e10s vs non-e10s Talos scores and an explanatory blog post are coming soon. Results in very raw form: http://pastebin.com/raw.php?i=D7fr2EtM.

Aaron Klotz created a tool for visualizing Windows attached input queues, as detailed on his blog. Attached input queues can cause odd problems like bug 1105386 where a page won’t load or render if the mouse is not moving.

With bug 1128768, we can now gather information about the amount of jank caused by different types of Flash content. This will be used to quantify the benefit of targeting Shumway at different types of Flash content.

Per-compartment CPU accounting (bug 674779) should land in the next week or two. Use it to report add-ons that use too much CPU or CPOWs (e.g. bug 1136923).

Shumway

The team’s current focus is playing Flash ads in Shumway instead of Flash. Using Shumway will reduce the number of Flash plugin instantiations, which is correlated with Flash deadlocks, and reduce page jank. Flash ads have also exploited Flash security bugs to install malware. Shumway should protect against many of those exploits.

Shumway now plays IMDb trailer videos on Nightly (bug 1137433) and substantial progress is being made on a verifiable sandboxed. An implementation of AS3’s meta-object protocol that should be pretty faithful to Flash’s an a new Shumway interpreter should land soon. This enables proper handling of SecurityDomains, a crucial Flash security feature.

Mozilla Firefox/Platform Engineering Update 38.3

This engineering update is also available on the Platform/2015-03-03 wiki.

Firefox Release Schedule

The next merge date is March 30, just four weeks away. Firefox 37 and 38 releases have been moved up one week to avoid conflicts with holidays. Firefox 38 will be the “spring thing” Firefox release and the next ESR, retiring ESR 31.

The new “Tracking Firefox” dashboard shows tracked bugs for the Aurora and Beta releases.

Upcoming Outages

The next Tree Closing Window is scheduled for Saturday, March 14.

MemShrink

A new Windows-only “address-space” memory reporter (bug 1134030) landed which may help debug Windows graphics and video memory usage. If you have other ideas on how to measure Windows graphics and video memory usage, contact Nicholas Nethercote.

Shumway

Shumway has now been enabled (in the Nightly channel only) for playing IMDb trailer videos. Shumway made some headlines a couple weeks ago (TechCrunch and CNET) when it was enabled for playing Amazon product tour videos.

DOM

Service Workers (bug 1059784) should land soon. (Famous last words.)

IndexedDB performance work will also land soon: bug 866846 will enable SQLite’s WAL journal and bug 1112702 will change transactions to be non-durable. These SQLite options favor performance over durability like Chrome and IE do. They do not increase the risk of database corruption.

Conferences

Mozilla’s Firefox OS team is at the MWC 2015 (Mobile World Congress) in Barcelona this week, announcing new Firefox OS partners, markets, and hardware form factors. Including sliders and flip phones!

Mozilla is also at GDC 2015 (Game Developer Conference) in San Francisco, where Epic Games and Mozilla announced Unreal Engine 4.7 support for HTML5/WebGL export.

Testing Add-on Compatibility With Multi-Process Firefox

“Electrolysis” (or “e10s” for short) is the project name for Mozilla’s multi-process Firefox. Sandboxing tabs into multiple processes will improve security and UI responsiveness. Firefox currently sandboxes plugins like Flash into a separate process, but sandboxing web content is more difficult because Firefox’s third-party add-ons were not designed for multiple processes. IE and Chrome use multiple processes today, but Google didn’t need to worry about add-on compatibility when designing Chrome’s multi-process sandbox because they didn’t have any. :)

And that’s where our Firefox Nightly testers come in! We can’t test every Firefox add-on ourselves. We’re asking for your help testing your favorite add-ons in Firefox Nightly’s multi-process mode. We’re tracking tested add-ons, those that work and those that need to be fixed, on the website arewee10syet.com (“Are We e10s Yet?”). Mozilla is hosting a QMO Testday on Friday August 1 where Mozilla QA and e10s developers will be available in Mozilla’s #testday IRC channel to answer questions.

To test an add-on:

  1. Install Firefox Nightly.
  2. Optional but recommended: create a new Firefox profile so you are testing the add-on without any other add-ons or old settings.
  3. Install the add-on you would like to test. See arewee10syet.com for some suggestions.
  4. e10s is disabled by default. Confirm that the add-on works as expected in Firefox Nightly before enabling e10s. You might find Firefox Nightly bugs that are not e10s’ fault. :)
  5. Now enable e10s by opening the about:config page and changing the browser.tabs.remote.autostart preference to true.
  6. Restart Firefox Nightly. When e10s is enabled, Firefox’s tab titles will be underlined. Tabs for special pages, like your home page or the new tab page, are not underlined, but tabs for most websites should be underlined.
  7. Confirm that the add-on still works as expected with e10s.
  8. To disable e10s, reset the browser.tabs.remote.autostart preference to false and restart Firefox.

Some e10s problems you might find include Firefox crashing or hanging. Add-ons that modify web page content, like Greasemonkey or AdBlock Plus, might appear to do nothing. But many add-ons will just work.

If the add-on works as expected, click the “it works” link on arewee10syet.com for that add-on or just email me so we can update our list of compatible add-ons.

If the add-on does not work as expected, click the add-on’s “Report bug” link on arewe10syet.com to file a bug report on Bugzilla. Please include the add-on’s name and version, steps to reproduce the problem, a description of what you expected to happen, and what actually happened. If Firefox crashed, include the most recent crash report IDs from about:crashes. If Firefox didn’t crash, copying the log messages from Firefox’s Browser Console (Tools menu > Web Developer menu > Browser Console menu item; not Web Console) to the bug might include useful debugging information.

JS Work Week 2014

Mozilla’s SpiderMonkey (JS) and Low-Level Tools engineering teams convened at Mozilla’s chilly Toronto office in March to plan our 2014 roadmap.

To start the week, we reviewed Mozilla’s 2014 organizational goals. If a Mozilla team is working on projects that do not advance the organization’s stated goals, then something is out of sync. The goals where the JS team can most effectively contribute are “Scale Firefox OS” (sell 10M Firefox OS phones) and “Get Firefox on a Growth Trajectory” (increase total users and hours of usage). Knowing that Mozilla’s plans to sell 10M Firefox OS phones helps us prioritize optimizations for Tarako (the $25 Firefox OS phone) over larger devices like Firefox OS tablets, TVs, or dishwashers.

Security was a hot topic after Mozilla’s recent beating in Pwn2Own 2014. Christian Holler (“decoder”) and Gary Kwong gave presentations on OOM and Windows fuzzing, respectively. Bill McCloskey discussed the current status of Electrolysis (e10s), Firefox’s multiprocess browser that will reduce UI jank and implement sandboxing of security exploits. e10s is currently available for testing in the Nightly channel; just select “File > New e10s Window” to open a new e10s window. (This works out of the box on OS X today, but requires an OMTC pref change on Windows and Linux.)

The ES6 spec is feature frozen and should be signed-off by the end of 2014. Jason Orendorff asked for help implementing remaining ES6 features like Modules and let/const scoping. Proposed improvements to Firefox’s web developer tools included live editing of code in the JS debugger and exposing JIT optimization feedback

Thinker Lee and Ting-Yuan Huang, from Mozilla’s Firefox OS team in Taipei, presented some of the challenges they’ve faced with Tarako, a Firefox OS phone with only 128 MB RAM. They’re using zram to compress unused memory pages instead of paging them to flash storage. Thinker and Ting-Yuan had suggestions for tuning SpiderMonkey’s GC to avoid problems where the GC runs in background apps or inadvertently touches compressed zram pages.

Till Schneidereit lead a brainstorming session about improving SpiderMonkey’s embedding API. Ideas included promoting SpiderMonkey as a scripting language solution for game engines (like 0 A.D.) or revisiting SpiderNode, a 2012 experiment to link Node.js with SpiderMonkey instead of V8. SpiderNode might be interesting for our testing or to Node developers that would like to use SpiderMonkey’s more extensive support for ES6 features or remote debugging tools. ES6 on the server doesn’t have the browser compatibility limitations that front-end web development does. The meeting notes and further discussion continued on the SpiderMonkey mailing list. New Mozilla contributor Sarat Adiraj soon posted his patches to revive SpiderNode in bug 1005411.

For the work week’s finale, Mozilla’s GC developers Terrence Cole, Steve Fink, and Jon Coppeard landed their generational garbage collector (GGC), a major redesign of SpiderMonkey’s GC. GGC will improve JS performance and lay the foundation for implementing a compacting GC to reduce JS memory usage later this year. GGC is riding the trains and should ship in Firefox 31 (July 2014).

Cloaking plugin names to limit browser fingerprinting in Firefox

Web analytics software often tracks people using a “fingerprint” of their browsers’ unique characteristics. Bumper stickers are a good analogy. If you see a blue Volkswagen with the same uncommon bumper stickers as a blue Volkswagen you saw yesterday, there is a very good chance it is the same person driving the same car. If you don’t want people to recognize your car, you can remove your bumper stickers or display the same bumper stickers seen on many other cars. The list of plugins and fonts installed on your computer are like bumper stickers on your browser.

For more technical details of fingerprinting, see Mozilla’s Fingerprinting wiki, the EFF’s Panopticlick demo, or the HTML Standard’s “hidden plugins” description.

I landed a fix for bug 757726 so Firefox 28 will “cloak” uncommon plugin names from navigator.plugins[] enumeration. This change does not disable any plugins; it just hides some plugin names.

If you find that a website no longer recognize your installed plugin when running Firefox 28, this is likely a side effect of bug 757726. Please file a new bug blocking bug 757726 so we can fix our whitelist of uncloaked plugin names or have a web compatibility evangelist reach out to the website author to fix their code.

This code change will reduce browser uniqueness by “cloaking” uncommon plugin names from navigator.plugins[] enumeration. If a website does not use the “Adobe Acrobat NPAPI Plug-in, Version 11.0.02″ plugin, why does it need to know that the “Adobe Acrobat NPAPI Plug-in, Version 11.0.02″ plugin is installed? If a website does need to know whether the plugin is installed or meets minimum version requirements, it can still check navigator.plugins[“Adobe Acrobat NPAPI Plug-in, Version 11.0.02″] or navigator.mimeTypes[“application/vnd.fdf”].enabledPlugin (to workaround problem plugins that short-sightedly include version numbers in their names).

For example, the following JavaScript reveals my installed plugins:

for (plugin of navigator.plugins) console.log(plugin.name);
“Shockwave Flash”
“QuickTime Plug-in 7.7.3″
“Default Browser Helper”
“Unity Player”
“Google Earth Plug-in”
“Silverlight Plug-In”
“Java Applet Plug-in”
“Adobe Acrobat NPAPI Plug-in, Version 11.0.02″
“WacomTabletPlugin”

navigator.plugins[“Unity Player”].name // get cloaked plugin by name
“Unity Player”

But with plugin cloaking, the same JavaScript will not reveal as much personally-identifying information about my browser:

for (plugin of navigator.plugins) console.log(plugin.name);
“Shockwave Flash”
“QuickTime Plug-in 7.7.3″
“Java Applet Plug-in”

navigator.plugins[“Unity Player”].name // get cloaked plugin by name
“Unity Player”

In theory, all plugin names could be cloaked because web content can query navigator.plugins[] by plugin name. Unfortunately, we could not cloak all plugin names because many popular websites check for Flash or QuickTime by enumerating navigator.plugins[] and comparing plugin names one by one, instead of just asking for navigator.plugins[“Shockwave Flash”] by name. These websites should be fixed.

The policy of which plugin names are uncloaked can be changed in the about:config pref “plugins.enumerable_names”. The pref’s value is a comma-separated list of plugin name prefixes (so the prefix “QuickTime” will match both “QuickTime Plug-in 6.4″ and “QuickTime Plug-in 7.7.3″). The default pref cloaks all plugin names except Flash, Shockwave (Director), Java, and QuickTime. To cloak all plugin names, set the pref to the empty string “”. To cloak no plugin names, set the pref to magic value “*”.

I started hacking on this patch in my spare time 13 months ago. I finally found some time to complete it. :)

Stacked Git: Mercurial-style Patch Queues for Git

Most Mozilla developers use Mercurial (hg) patch queues to organize patches and revise them based on feedback from Bugzilla code reviews. I had not found a similar git workflow that was nearly as convenient until Stacked Git. Suggestions to use git rebase --interactive and temporary branches were not as nimble as being able to push and pop patches on a branch.

Stacked Git (stg) is a git porcelain script that adds support for hg-style patch queues. stg’s patch queues augment, not replace, git’s feature branches. When you start a patch queue for a branch, stg creates a corresponding “my_branch.stgit” branch. Your patch queue operations are recorded in the “my_branch.stgit” branch, but your “my_branch” has commits for each of your applied patches. OSX users can use Homebrew to install Stacked Git: `brew install stgit`

Mercurial command Stacked Git command
hg qnew my_patch stg new my_patch
hg qrefresh stg refresh
hg qseries stg series
hg qrename my_new_patch stg rename my_new_patch
hg qrefresh --edit stg edit
hg qfinish stg patches are already commits

Here is an example GitHub workflow:

# Create a new branch “my_branch” to fix a bug:
$ git checkout -b my_branch master

# Configure your branch for stg:
$ stg init

# … Edit some files to fix the bug

# Create a new patch (which also creates a temporary commit):
$ stg new my_patch

# Push your fix to GitHub:
$ git push my_github_remote my_branch

# … Submit a pull request on GitHub
# … Edit some files to revise my fix based on review feedback

# Update my patch:
$ stg refresh

# Push my revised fix to GitHub:
$ git push --force my_github_remote my_branch

# Now that my_branch has been merged on GitHub, delete local branches, my_branch and my_branch.stgit:
$ git checkout master
$ stg branch --cleanup my_branch
$ git branch -d my_branch

H.264 video in Firefox for Android


Firefox for Android
has expanded its HTML5 video capabilities to include H.264 video playback. Web developers have been using Adobe Flash to play H.264 video on Firefox for Android, but Adobe no longer supports Flash for Android. Mozilla needed a new solution, so Firefox now uses Android’s “Stagefright” library to access hardware video decoders. The challenges posed by H.264 patents and royalties have been documented elsewhere.

Supported devices

Firefox currently supports H.264 playback on any device running Android 4.1 (Jelly Bean) and any Samsung device running Android 4.0 (Ice Cream Sandwich). We have temporarily blocked non-Samsung devices running Ice Cream Sandwich until we can fix or workaround some bugs. Support for Gingerbread and Honeycomb devices is planned for a later release (Bug 787228).

To test whether Firefox supports H.264 on your device, try playing this “Big Buck Bunny” video.

Testing H.264

If your device is not supported yet, you can manually enable H.264 for testing. Enter about:config in Firefox for Android’s address bar, then search for “stagefright”. Toggle the “stagefright.force-enabled” preference to true. H.264 should work on most Ice Cream Sandwich devices, but Gingerbread and Honeycomb devices will probably crash.

If Firefox does not recognize your hardware decoder, it will use a safer (but slower) software decoder. Daring users can manually enable hardware decoding. Enter about:config as described above and search for “stagefright”. To force hardware video decoding, change the “media.stagefright.omxcodec.flags” preference to 16. The default value is 0, which will try the hardware decoder and fall back to the software decoder if there are problems (Bug 797225). The most likely problems you will encounter are videos with green lines or crashes.

Giving feedback/reporting bugs

If you find any video bugs, please file a bug report here so we can fix it! Please include your device model, Android OS version, the URL of the video, and any about:config preferences you have changed. Log files collected from aLogcat or adb logcat are also very helpful.

(Cross-posted on the Mozilla Hacks blog: H.264 video in Firefox for Android.)