Tag Archives: mochitest

June 9, 2015 · 12:32 pm

Please join me in welcoming the DX Team

A few weeks ago, I posted a call out for people to reach out and commit to participate for 8+ weeks.  There were two projects and one of them was Developer Experience.  Since then we have had some great interest, there are 5 awesome contributors participating (sorted by irc nicks).

BYK – I met BYK 3+ years ago on IRC- he is a great person and very ambitious.  As a more senior developer he will be focused primarily on improving interactions with mach.  While there are a lot of little things to make mach better, BYK proposed a system to collect information about how mach is used.

gma_fav – I met gma_fav on IRC when she heard about the program.  She has a lot of energy, seems very detail oriented, asks good questions, and brings fresh ideas to the team!  She is a graduate of the Ascend project and is looking to continue her growth in development and open source.  Her primary focus will be on the interface to try server (think the try chooser page, extension, and taking other experiments further).

kaustabh93 – I met Kaustabh on IRC about a year ago and since then he has been a consistent friend and hacker.  He attends university.  In fact I do owe him credit for large portions of alert manager.  While working on this team, he will be focused on making run-by-dir a reality.  There are two parts: getting the tests to run green, and reducing the overhead of the harness.

sehgalvibhor – I met Vibhor on IRC about 2 weeks ago.  He was excited about the possibility of working on this project and jumped right in.  Like Kaustabh, he is a student who is just finishing up exams this week.  His primary focus this summer will be working in a similar role to Stanley in making our test harnesses act the same and more useful.

stanley – When this program was announced Stanley was the first person to ping me on IRC.  I have found him to be very organized, a pleasure to chat with and he understands code quite well.  Coding and open source are both new things to Stanley and we have the opportunity to give him a great view of it.  Stanley will be focusing on making the commands we have for running tests via mach easier to use and more unified between harnesses.

Personally I am looking forward to seeing the ambition folks have translate into great solutions, learning more about each person, and sharing with Mozilla as a whole the great work they are doing.

Take a few moments to say hi to them online.

1 Comment

Filed under Community, testdev

Tagged as , , , ,

April 14, 2014 · 11:39 am

browser-chrome is greener and in many chunks

On Friday we rolled out a big change to split up our browser-chrome tests.  It started out as a great idea to split the devtools out into their own suite, then after testing, we ended up chunking the remaining browser chrome tests into 3 chunks.

No more 200 minute wait times, in fact we probably are running too many chunks.  A lot of heavy lifting took place, a lot of it in releng from Armen and Ben, and much work from Gavin and RyanVM who pushed hard and proposed great ideas to see this through.

What is next?

There are a few more test cases to fix and to get all these changes on Aurora.  We have more work we want to do (lower priority) on running the tests differently to help isolate issues where one test affects another test.

In the next few weeks I want to put together a list of projects and bugs that we can work on to make our tests more useful and reliable.  Stay tuned!

 

1 Comment

Filed under testdev

Tagged as , , , ,

April 8, 2014 · 3:54 pm

polishing browser-chrome – coming to a branch near you soon

The last 2 weeks I have gone head first into a world of resolving some issues with our mochitest browser-chrome tests with RyanVM, Armen, and the help of Gavin and many developers who are fixing problems left and right.

There are 3 projects I have been focusing on:

1) Moving our Linux debug browser chrome tests off our old fedora slaves in a datacenter and running them on ec2 slave instances, in bug 987892.

These are live and green on all Firefox 29, 30, and 31 trees!  More work is needed for Firefox-28 and ESR-24 which should be wrapped up this week.  Next week we can stop running all linux unittests on fedora slaves.

2) Splitting all the developer tools tests out of the browser-chrome suite into their own suite in bug 984930.

browser-chrome tests have been a thorn in the side of the sheriff team for many months.  More and more the rapidly growing features and tests of developer tools have been causing the entire browser-chrome suite to fail, in cases of debug to run for hours.  Splitting this out gives us a small shield of isolation.  In fact, we have this running well on Cedar, we are pushing hard to have this rolled out to our production and development branches by the end of this week!

3) Splitting the remaining browser chrome tests into 3 chunks, in bug 819963.

Just like the developer tools, we have been running browser-chrome in 3 chunks on Cedar.  With just 7 tests disabled, we are very green and consistently green. 

 

 

While there are a lot of other changes going on under the hood, what will be seen by next week on your favorite branch of Firefox will be:

  • ‘dt’ jobs for opt, and ‘dt1’, ‘dt2’, ‘dt3’ jobs for debug
  • ‘bc’ job will turn into ‘bc1’, ‘bc2’, ‘bc3’
  • much faster turnaround times on bc tests (62 minutes is the slowest right now, the rest are averaging ~20 minutes/job)
  • less random orange cluttering up results

 

Leave a comment

Filed under Uncategorized

Tagged as , , ,

March 13, 2014 · 10:58 am

mochitests and manifests

Of all the tests that are run on tbpl, mochitests are the last ones to receive manifests.  As of this morning, we have landed all the changes that we can to have all our tests defined in mochitest.ini files and have removed the entries in b2g*.json, by putting entries in the appropriate mochitest.ini files.

Ahal, has done a good job of outlining what this means for b2g in his post.  As mentioned there, this work was done by a dedicated community member :vaibhav1994 as he continues to write patches, investigate failures, and repeat until success.

For those interested in the next steps, we are looking forward to removing our build time filtering and start filtering tests at runtime.  This work is being done by billm in bug 938019.  Once that is landed we can start querying which tests are enabled/disabled per platform and track that over time!

1 Comment

Filed under Uncategorized

Tagged as , , ,

October 5, 2011 · 3:35 pm

Android automation is becoming more stable ~7% failure rate

At Mozilla we have made our unit testing on android devices to be as important as desktop testing. Earlier today I was asked how do we measure this and what is our definition of success. The obvious answer is no failures except for code that breaks a test, but reality is something where we allow for random failures and infrastructure failures. Our current goal is 5%

So what are these acceptable failures and what does 5% really mean. Failures can happen when we have tests which fail randomly, usually poorly written tests or tests which have been written a long time ago and hacked to work in todays environment. This doesn’t mean any test that fails is a problem, it could be a previous test that changes a Firefox preference on accident. For Android testing, this currently means the browser failed to launch and load the test webpage properly or it crashed in the middle of the test. Other failures are the device losing connectivity, our host machine having hiccups, the network going down, sdcard failures, and many other problems. With our current state of testing this mostly falls into the category of losing connectivity to the device. For infrastructure problems they are indicated as Red or Purple and for test related problems they are Orange.

I took at a look at the last 10 runs on mozilla-central (where we build Firefox nightlies from) and built this little graph:

Firefox Android Failures

Firefox Android Failures

Here you can see that our tests are causing 6.67% of the failures and 12.33% of the time we can expect a failure on Android.

We have another branch called mozilla-inbound (we merge this into mozilla-central regularly) where most of the latest changes get checked in.  I did the same thing here:

mozilla-inbound Android Failures

mozilla-inbound Android Failures

Here you can see that our tests are causing 7.77% of the failures and 9.89% of the time we can expect a failure on Android.

This is only a small sample of the tests, but it should give you a good idea of where we are.

3 Comments

Filed under testdev

Tagged as , , , , , ,

November 23, 2010 · 10:08 am

Automation found its first regression for Fennec!

Yesterday I noticed our Mochitest-1 test was orange (since Nov 17th…still need to train others to watch the results), so I spent a few minutes looking into it and it turned out to be a regression (which was promptly fixed by the mobile guys!)  I know automation has uncovered errors in the past, but this is the first time we have had automation that was truly green for a week (we just turned it on) and then turned orange as the result of a unfriendly patch.

 

1 Comment

Filed under qa, testdev

Tagged as , , ,

October 1, 2010 · 1:04 pm

mochikit.jar changes are in mozilla central

Last night we landed the final patches to make mochikit.jar a reality.  This started out as a bug where we would package all the mochikit harness + chrome tests into a single .jar file and then on a remote system copy that to the application directory and run the tests locally.  It ended up being much more than that, let me explain some of the changes that have taken place.

why change all of this?

In order to test remotely (on mobile devices such as windows mobile and android) where there are not options to run tools and python scripts, we need to put everything in the browser that it needs and launch the browser remotely.  The solution for tests that are not accessible over the network is to run them from local files.

what is mochikit.jar?

Mochikit.jar is an extension that is installed in the profile and contains all the core files that mochitest (plain, chrome, browser-chrome, a11y) needs to run in a browser.  This doesn’t contain any of the external tools such as ssltunnel and python scripts to set up a webserver.  When you do a build, you will see a new directory in $(objdir)/_tests/testing/mochitest called mochijar.  Feel free to poke around there.  As a standalone application all chrome://mochikit/content calls will use this extension, not a pointer to the local file system.  The original intention of mochkit.jar was to include tests data, but we found that to create an extension using jar.mn we needed a concrete list of files and that was not reasonable to do for our test files.  So we created tests.jar.

what is tests.jar?

tests.jar is the actual test data for browser-chrome, chrome, and a11y tests.  These are all tests that are not straightforward to access remotely over http, so we are running these locally out of a .jar file.  tests.jar is only created when you do a ‘make package-tests’ and ends up in the root of the mochitest directory as tests.jar.  If the harness finds this file, it copies it to the profile and generates a .manifest file for the .jar file, otherwise we generate a plain .manifest file to point to the filesytem.  Finally we dynamically registers tests.manifest from the profile.  Now all your tests will have a chrome://mochitests/content instead of chrome://mochikit/content.

What else changed?

A lot of tests had to change to work with this because we had hardcoded chrome://mochikit/content references in our test code and data.  It is fine to have that in there for references to the harness and core utilities, but to reference a local piece of test data, it was hardcoded and didn’t need to be.  A few tests required some more difficult work where we had to extract files temporarily to a temp folder in the profile and reference them with a file path.

what do I need to do when writing new tests?

please don’t cut and paste code then change it to reference a data, utility, or other uri that has chrome://mochikit/content/ in it.  If you need to access a file with the full URI or as a file path, here are some tips:

* a mochitest-chrome test that needs to reference a file in the same directory or subdir:
let chromeDir = getRootDirectory(window.location.href);

* a browser-chrome test that needs to reference a file in the same directory or subdir:
//NOTE: gTestPath is set because window.location.href is not always set on browser-chrome tests
let chromeDir = getRootDirectory(gTestPath);

* extracting files to temp and accessing them

  let rootDir = getRootDirectory(gTestPath);
  let jar = getJar(rootDir);
  if (jar) {
    let tmpdir = extractJarToTmp(jar);
    rootDir = "file://" + tmpdir.path + '/';
  }
  loader.loadSubScript(rootDir + "privacypane_tests.js", this);

Leave a comment

Filed under general, testdev

Tagged as , , , ,

July 27, 2010 · 5:07 pm

types of data we care about in a manifest

This is a bit controversial (similar to “what OS do you run”), but I want to start outlining what I find as useful metadata to categorize tests with.

DATA:

Currently with Reftest, we have a sandbox that provides data the manifest files can use as conditional options.  The majority of the sandbox items used are:

  • platform: MOZ_WIDGET_TOOLKIT (cocoa, windows, gtk2, qt)
  • platform: xulrunner.OS, XPCOMABI (if “”, we are on ARM)
  • environ:  haveTestPlugin, isDebugBuild, windowsDefaultTheme
  • config:    getBoolPref, getIntPref, nativeThemePref (could be a getBoolPref)

This is the type of information that a large portion of our tests care about.  Most of these options are somehow coded into mochitests as well (through getPref calls, checking the OS, or in Makefile conditions).  I would like to figure out how to add this type of data:

  • orange:   list of conditions this is random-orange on (but should pass)
  • fails:        list of conditions this is expected to fail on
  • e10s:       what service is used to cause this to fail under e10s
  • fennec:   does this fail when run on fennec, which platforms, what versions
  • remote:   does this fail when running tests remotely (required for android, originally for winmo)
  • results:    number of failures (if orange or fails above) and number of todo
  • runtime:  expected time to run this test (important on mobile)
  • product:  product name and version
  • future:     anything we find valuable in the future!

I can think of many ways to add this into the Reftest format or create a new format.  Looking at this data a but further, it really is not adding a lot of new information.  If we take the assumption that all tests are expected to pass in all configurations, any value assigned to a new piece of data would indicate that it fails under that given condition (or list of conditions).  As our supported platforms, configurations, and products grow, we will have a much greater need for this additional metadata.

INTEGRATION:

I would like to make all data pieces as tags vs raw conditions (Reftest does them like C conditions.)  This would allow much greater flexibility and adding data that doesn’t necessarily turn on/off a test.  For example, lets say a test is a random-orange for Firefox 1.9.1 (not 1.9.2), fails on Fennec Maemo 1.1 only, is orange on remote testing on Fennec android and currently is broken by e10s.  We could easily add those conditions to a list:

fails-if(OS==Maemo) fails-if(e10s==nsIPrefService) random-if(product=Firefox && xr.version==1.9.1) random-if(os=Android && remote==true) test_bitrot.html

So this is doable (please disregard any misused fails-if, random-if statements) and wouldn’t be too hard to add into a reftest.list style format for Mochitest (and even Reftest.)  Initially I thought it would be really cool to just run fails-if, random-if or skip-if statements with a small tweak to the command line.  This would give us the flexibility to turn on and off tests easier, but I realized that it would turn on/off all tests related to the condition.  I think a small adjustment in the format might allow for tags and we could tweak a run in the future with little work.  One example might be like:

fails(os=Maemo;e10s=nsIPrefService,nsIDOMGeoGeolocation) random(product=Firefox&xr.version=1.9.1;os=Android&remote=true) test_bitrot.html

This example is a minor change (which might not be needed), but helps simplify the syntax and keep in mind the idea of tags.  The backend details would need to be changed to support a ‘toggle’ of a tag in either scenario.  Maybe we just want to run e10s tests.  We can find all tests that have a e10s=nsiPrefService tag inside a fails tag block and just run those specific tests while maintaining all the other rules (skip on a certain OS or product).

There are still questions if the Reftest format is the right format for Mochitest.  It has a lot of weight since it works so well for so many of our tests.

3 Comments

Filed under testdev

Tagged as ,

May 24, 2010 · 10:45 pm

tests that require privileged access

I have been working on a project to get mochitests running on a build of Fennec + electrolysis.  In general, you can follow along in bug 567417.

One of the large TODO items in getting the tests to run is actually fixing the tests which use UniversalXPConnect.  So my approach was to grep through a mochitest tests/ directory for @mozilla and parse it out.  With a few corner cases, this resulted in a full list of services we utilize from our tests (here is a sorted list by frequency 76 total services.)  Cool, but that didn’t seem useful enough.  Then I took my work that I have done for filtering (the json file) and cross referenced that with my original list of tests that use UniversalXPConnect.

Now I have a list of 59 services which all should pass in Fennec (a mozilla-central build from 2 weeks ago on n900) along with the first filename of the test which utilizes that services!

What else would be useful?

Leave a comment

Filed under testdev

Tagged as , , , ,

May 13, 2010 · 12:56 pm

accessing privileged content from a normal webpage, request for example

One project which I am working on is getting mochitests to run in fennec + electrolysis.  Why this is a project is we don’t allow web pages to access privileged information in the browser anymore.  Mochitests in the simplest form use file logging and event generation.

The communication channel that is available between the ‘chrome’ and ‘content’ process is the messageManager protocol.  There are even great examples of using this in the unit tests for ipc.  Unfortunately I have not been able to load a normal web page and allow for my extension which used messageManager calls to interact.

I think what would be nice to see is a real end to end example of an extension that would demonstrate functionality on any given webpage.  This would be helpful to all addon authors, not just me:)  If I figure this out, I will update with a new blog post.

1 Comment

Filed under testdev

Tagged as , , , , , ,