I got to release the new update to the Add-on SDK this last Tuesday. Jeff Griffiths was on a vacation leading up to the release, so the tasks he usually does for the releases got split up between other people on the team.
We tweaked the usual schedule a bit to make up for the fact that we were all doing some things that we usually didn't do, though I think that we should be doing it this way from now on, if possible, because it makes the actual release day a lot less hectic.
On Tuesday, September 11, I tagged and pushed the second SDK 1.10 release candidate build.
Dave Townsend blessed it as our final candidate build the next day, and I proceeded to tag and push the final SDK 1.10 to the servers. After merging 1.10 to the release branch in the repository, I ran the AMO validator hashing script and issued a pull request to AMO so that AMO would accept addons built using SDK 1.10 as valid extensions.
I then posted to the bug that tracked getting the Addon Builder site updated to using SDK 1.10 with a link to the final 1.10 tag in our repository.
Working with the AMO people, we made sure that the updates to the validator hashes and the 1.10 documentation were scheduled to be pushed live on Thursday September 13 so everything would be ready for the official release on the following Tuesday.
I spent some time on Wednesday and Thursday talking with the Builder people to make sure that the changes to Builder were set to be pushed in a timely fashion. We also pushed 1.10 to the Builder development server to make sure that everything was working correctly before pushing it to the production server on Tuesday.
On Friday, I built an empty extension using the final SDK 1.10 bits and uploaded it to the AMO validator to make sure that the validator changes had been pushed live (and actually worked), and everything was working fine for that. I also verified that the Builder-dev instance had 1.10 added and it was usable, and then tried to make sure that everything was properly set for Tuesday's push to the production server.
On Tuesday September 18, after double-checking with Dave, Irakli and Will that we were good to proceed with the actual release, I set the correct permissions for the 1.10 zip and tarball files so that people could download them, updated the "latest" redirects for those files, and Will announced the release.
I then checked the production Builder site to see if 1.10 was pushed live, but it wasn't. Pinging people on IRC didn't really get much response. (It was late in the day for the Americas at this point, and some of the relevant people for Builder are over in Europe, so it was the middle of the night for them.) I left it alone for the night, choosing to pester about it earlier on the following day.
On Wednesday September 19, I resumed pestering, and found that the people in IT that have permissions to push to production respond much quicker having bugs assigned to them for the various tasks you need them to do than they do when you just ping on IRC or CC them to the relevant bugs. We were able to get 1.10 pushed to production Builder later that day, and all was right with the world again.
Except that after Jeff returned from his vacation, he discovered that Builder believed SDK 1.10 was older than SDK 1.2 (because 1.2 minus 1.10 is a positive number, basically), so the various checks in Builder for displaying only the XX most recent SDK releases in the dropdown were suddenly displaying a lot more old releases than they should have been. This issue was discovered on Friday, and there's an implied "no live pushes" rule for Fridays (since then you have to deal with fallout from the push over the weekend when less people are around), so the fix for it wasn't pushed until Monday September 24. Which really isn't a big deal, since Builder still was defaulting to 1.10 for new addons.
Things I learned from this:
- Getting the final release candidate built, tested, and blessed on the Tuesday/Wednesday of the week prior to the actual release is really helpful. It lets us get the validator and Builder changes set up long before the deadlines for each of them are close. We should do it like that from now on.
- The documentation for getting the amo-validator and Builder repositories up and running really assumes that the reader knows what they're doing, and has a computer with most/all of the prerequisites already installed. Doing some of these tasks for the first time this cycle was a learning experience on its own.
- Builder is still the weak point in our release process. As a somewhat-external group from the core Jetpack team, it's harder to get things coordinated (and we can't just push what we need pushed when we want it to be pushed). We need to get much more coordination between everyone involved, much earlier in each release cycle.
What can we do?
- Make sure the amo-validator and flightdeck repositories have correct documentation for an outsider to know what needs to be done to update them correctly.
- Assign bugs to specific people in IT and give a description of what needs to be pushed or run when you want them to push/run something for you. Don't just file a bug in their product/component in Bugzilla or CC them on the bug and assume that they know what to do and when to do it. After you do that, then you can ping them on IRC to see if they're available to do things immediately.
- In the week prior to the actual release, talk with the people who are in charge of doing things we need done, to make sure that they know that the release is coming up. Maybe they can do the actual assigning of bugs from the previous bullet point for us to make sure that the assigned person will actually be around at the times when we need the tasks done.
- Long term, we need to continue with the plans to land the SDK into Firefox so that Builder could just become a simple IDE and repository backend for people developing SDK-based addons. That way we don't need to go through with all of this hassle every six weeks.