Wednesday, September 28, 2016

[NEW] Added build status updates - Usability improvements for Firefox automation initiative - Status update #6

[NEW] Starting on this newsletter we will start giving you build automation improvements since they help with the end to end time of your pushes

On this update we will look at the progress made in the last two weeks.

A reminder that this quarter’s main focus is on:
  • Debugging tests on interactive workers (only Linux on TaskCluster)
  • Improve end to end times on Try (Thunder Try project)

For all bugs and priorities you can check out the project management page for it:

Status update:
Debugging tests on interactive workers
---------------------------------------------------

Accomplished recently:
  • Fixed regression that broke the interactive wizard
  • Support for Android reftests landed

Upcoming:
  • Support for Android xpcshell
  • Video demonstration


Thunder Try - Improve end to end times on try
---------------------------------------------

Project #1 - Artifact builds on automation
##########################################

Accomplished recently:
  • Windows and Mac artifact builds are soon to land
  • |mach try| now supports --artifact option
  • Compiled-code tests jobs error-out early when run with --artifact on try

Upcoming:
  • Windows and Mac artifact builds available on Try
  • Fix triggering of test jobs on Buildbot with artifact build

Project #2 - S3 Cloud Compiler Cache
####################################

Nothing new in this edition.

Project #3 - Metrics
####################

Accomplished recently:

  • Drill-down charts:

  • Which lead to a detailed view:

  • With optional wait times included (missing 10% outliers, so looks almost the same):


Upcoming:
  • Iron out interactivity bugs
  • Show outliers
  • Post these (static) pages to my people page
  • Fix ActiveData production to handle these queries (I am currently using a development version of ActiveData, but that version has some nasty anomalies)

Project #4 - Build automation improvements
##########################################
Upcoming:


Project #5 - Run Web platform tests from the source checkout
############################################################
Accomplished recently:
  • WPT is now running from the source checkout in automation

Upcoming:
  • There are still parts in automation relying on a test zip. Next steps is to minimize those so you can get a loner, pull any revision from any repo, and test WPT changes in an environment that is exactly what the automation tests run in.

Other
#####
  • Bug 1300812 - Make Mozharness downloads and unpacks actions handle better intermittent S3/EC2 issues
    • This adds retry logic to reduce intermittent oranges


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Tuesday, September 13, 2016

Increasing test coverage

Last quarter I spent some time increasing mozci's test coverage. Here are some notes I took to help me remember in the future how to do it.

Here's some of what I did:
  • Read Python's page about increasing test coverage
    • I wanted to learn what core Python recommends
    • Tthey recommend is using coverage.py
  • Quick start with coverage.py
    • "coverage run --source=mozci -m py.test test" to gather data
    • "coverage html" to generate an html report
    • "/path/to/firefox firefox htmlcov/index.html" to see the report
  • NOTE: We have coverage reports from automation in coveralls.io
  • If you find code that needs to be ignored, read this.
    • Use "# pragma: no cover" in specific lines
    • You can also create rules of exclusion
  • Once you get closer to 100% you might want to consider to increase branch coverage instead of line coverage
    • Read more in here.
  • Once you pick a module to increase coverage
    • Keep making changes until you run "coverage run" and "coverage html".
    • Reload the html page to see the new results
After some work on this, I realized that my preferred place to improve tests is focusing on the simplest unit tests. I say this since integration tests do require proper work and thinking how to properly test them rather than *just* increasing coverage for the sake of it.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Tuesday, August 02, 2016

Usability improvements for Firefox automation initiative - Status update #2

On this update, we will look at the progress made since our initial update.

A reminder that this quarter’s main focus is on:
  • Debugging tests on interactive workers (only Linux on TaskCluster)
  • Improve end to end times on Try (Thunder Try project)

For all bugs and priorities you can check out the project management page for it:

Debugging tests on interactive workers

Accomplished recently:
  • Bug 1285582 - Fixed Xvfb startup issue
  • Bug 1288827 - Improved mochitest UX (no longer need --appname, paths normalized)
  • Bug 1289879 - Uses mozharness venv if available

Upcoming:
  • Support for smaller test harnesses (Cpp, Mn, wpt, etc)
  • Improved one-click-loaner UX


Thunder Try - Improve end to end times on try

Project #1 - Artifact builds on automation

No news for this edition and probably the next one.


Project #2 - S3 Cloud Compiler Cache

Accomplished recently:
  • Working on testing sccache re-write on Try
  • More news on following update


Project #3 - Metrics

Accomplished recently:
  • Bug 1242017 - Metrics team will configure ingestion point into Telemetry

Upcoming:
  • Bug 1258861 - Working on underlying data model at the moment:


Other
  • Bug 1287604 - Experiment with different AWS instance types for TC linux64 builds
    • Some initial experiments have shown we can shave 20 minutes off an average linux64 build by using more powerful AWS instances, with a reasonable cost tradeoff. We’ll start the work of migrating to these new instances soon.
  • Bug 1272083 - Downloading and unzipping should be performed as data is received
  • Bug 1286336 - Improve interaction of automation with version control
    • Buildbot AMIs now seeded with mozilla-unified repo (Bug 1232442)
    • TaskCluster decision and various lint/test tasks now use `hg robustcheckout` and share caches more optimally (Bug 1247168)
      • Flake8 tasks now complete in as little as 9s (~3m before)
      • Decision tasks now complete in <60s average="" font="" on="">
    • Some TaskCluster tasks now share VCS checkouts on Try (Bug 1289643)
      • Tasks will complete faster on Try due to not having to perform full VCS checkout

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1247168


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.