There are approximately 30 to 40 core developers of XBMC, of whom probably 15 are directly involved in the release of any particular version of XBMC. Each operating system has a boss whose job it is to say that XBMC is working in his environment. Above the OS boss is the Release Manager, who ensures that no more code is entered into XBMC once we hit our target release date and who is primarily in charge of ensuring that all the coding changes that need to be made for the update are made, as well as limiting code changes (made by something called PRs) to fixes, rather than features.

There are also packaging managers. These are people who focus on making sure that XBMC is compiled and built correctly and then placed in the proper folder for easy access to Windows, OSX, iOS, and AppleTV builds. One person maintains the Linux PPA (aka the Linux repository that allows individuals to easily download and update). One person maintains the XBMCbuntu package.

Other people manage scripts, plugins, and skins for the addon repository, as well as flagging compliance with our API and json-rpc calls.

Are all the ducks in a row yet? … Nope.

One person heads the wiki for installation instructions and FAQs about all the new updates. And finally, one person makes sure all the ducks are in a row and writes up the announcement.

That last person is me, and boy do I have a story to tell.

 

 

Taking a Step Back

If it isn’t already clear, XBMC is BIG. Like, really, really big. There are actually more lines of code in XBMC than in the entire Android kernel. Admittedly, quite a bit of that is probably related to 10 years of history with limited pruning, but consider this: XBMC can act as a full OS front-end replacement. When paired with a fairly limited number of Linux packages, it literally acts as an Operating System (in the form of OpenELEC). On every platform, it has the ability to launch programs. On Android, it appears to launch programs exactly the same as the standard Android launcher.

Given the sheer size, the fact that we have several people who understand the platform well enough to act as Release Managers is absolutely astounding. We have been lucky at least once with this release as long-time XBMC dev Spiff agreed to helm the position for Beta1. For personal reasons, it appears he won’t be able to head up Beta2, but almost immediately after informing the team, several other individuals with the requisite knowledge offered to step forward.

The release manager position is a brutal one. The issue isn’t simply one of knowing when XBMC is ready for Beta. It’s also having to make decisions about what code is a fix and what isn’t, and then taking a daily stand against developers who disagree. Telling people no, over and over, particularly when they’re convinced they are right (i.e. the entire XBMC development team… ♡ you guys!) is exhausting and has been known to lead to quite a few heated arguments.

But we have individuals who step up, drink a LOT after the release, and somehow manage to get on with their lives. More than anyone else, this story is dedicated to them (and particularly to Spiff, since this story is all about how we hit Frodo Beta 1).

The Decision to Feature Freeze

Because XBMC is not part of a larger corporate hierarchy, there is no decision made by higher ups to hit a target release date. Usually, we just keep working on the software and adding features until somebody says, “Well, that seems like enough.”

With Frodo, things were a bit different. At the end of Eden, we already had a tenuous list of items that we all agreed we’d like to see in Frodo before releasing. These were features that had been requested for quite a long time. They were:

  • HD Audio
  • PVR
  • h.264 10-bit decoding
  • Jim Carroll’s code generation work to improve the communication of core XBMC with addons (primarily an internal request)

The first item made it in almost immediately after Eden was released. HD Audio had been ready for quite some time. The Team was hesitant to pull it in, primarily because it was practically guaranteed to break things since audio touches so many other pieces of the project.

The second item to make it in was 10-bit decoding. For the most part, that was simply an ffmpeg update.*

*I say “simply,” but let me assure you, an ffmpeg update can be a truly painful experience, given the degree to which the code is patched to fit in XBMC.

Third in was PVR. The primary holdup for PVR, for quite some time now, has been the lack of binary addon support. As you may know, a major goal of XBMC is always to follow the KISS principle. Though we had planned, long ago, not to become full time developers of any PVR software, we wanted to make the addition of a PVR source as simple and painless as possible. To make that happen, XBMC needed to be able to simply install the PVR software itself. We call the installation by XBMC of full applications “binary addon support.”

The ability to install and launch full programs is an inconsistent one between XBMC platforms, and figuring out how to make that ability consistent is something of a coding and logistical nightmare.* Eventually, with Frodo, we decided we weren’t going to let the nightmare of binary addon support prevent us from pushing PVR support. In theory in the future you will be able to install full packages of other open source software, like the PVR server TVheadend, so you don’t have to look elsewhere to get your PVR support. Right now, because we don’t support binary addons, you have to install and configure TVheadend yourself. It’s not ideal, but at least users now have the option.

*This, by the way, is the reason XBMC doesn’t self-update. It would need to launch an updater across multiple platforms. Since it CAN’T do that right now, the only instances of self-update are ones externally provided.

Nope, still don’t get it.

Fourth on the listed of necessary for Frodo items was Jim Carroll’s code generation work. I’d try to explain that stuff to you, but I barely understand it myself (read: honestly, don’t understand it), so I’m pretty useless on the topic. Anyway, that final piece of the puzzle was added a mere month ago in October.

And with that, we entered Beta phase.

 

 

A False Start

I already spoke about the unpleasant job of Release Manager. I won’t rehash that bit, mostly because the time from feature freeze to a few days before beta release is pretty boring. Devs read bug reports, look at bugs, and then fix them. There’s very little drama in that, beyond the drama already described. The drama this year has been even more limited than normal. People are used to spending 2/3rds of every month purely working on personal things or bug fixes. In many ways, Frodo is likely the single most stable Beta1 of XBMC in the long history of the organization,* thanks in large part to the Team’s dedication to the monthly merge cycle.

*For the sake of clarification, I am referring to stability in the core code. We’ve already discovered some major errors in both XBMC for Windows and XBMCbuntu that were the result of an ugly build process. Simply rebuilding those versions of Beta1 resolved the errors.

Seven days before the unofficial due date of Beta1 and a mere three days after feature freeze, Jonathan Marshall added Frodo labels to all github issues and PRs that he believed were Frodo material.

Five days before the unofficial due date, Spiff wrote, “ping. 5 days to beta 1. what’s the status on platform foo?” In XBMC lingo (possibly all dev lingo, I’m not sure) this means, approximately, “Hey guys. We’ve got 5 days. I need each of the platform bosses to give me the thumbs up or down on the status of Linux, OSX, Windows, iOS, and Android. Is each platform free enough of bugs to put in beta?”

Montellese and Martijn gave the thumbs up for Windows. Memphiz and Scott Davilla gave the thumbs up on OSX/iOS/atv2.  Spiff himself gave the thumbs up on Linux. Cory Fields put up the red flag on Android, because it was guaranteed to crash any and all Android devices that didn’t include the NEON extension upon launch. After a quick huddle (primarily between Cory and myself, as we were the ones most likely to deal with the resulting furor and chaos), we decided to push releasing the first Android apk to a later date, to make sure we didn’t crash half the handsets in America.*

*That’s right, Android fans. I’m about 50% of the reason you still have to look to third parties for your XBMC apks. My bad! :)

The only real issue was the Linux ppa. As I said, Spiff gave the go ahead on Linux itself, but with each platform, the platform chief go-ahead is only half the equation. The individual in charge of building and hosting also has to give the green light.  The problem was, we literally DIDN’T HAVE a chief to build and host a Linux build on our ppa.

For those unfamiliar with how installing applications works in Linux, it’s fairly simple. Picture the Apple Appstore. Now realize that that is how Ubuntu has been installing most applications since the early 2000s. Maybe even earlier. Possibly since the 90s.* Essentially, anybody can make their own Linux Appstore and Linux users can download from that store. Except in Linux-land, this store is called a “ppa.”

*As a guy who tends not to use Linux, it’s often depressing to me to realize how far behind my platforms of choice are compared to Linux (or, at the very least, Debian).

Sadly, not the wizard we were looking for.

The problem is, essentially nobody on the team had any experience setting up a ppa. We’d gotten somebody else to do it for Eden, and he was fairly busy this go around. So Piethein (one of our XBMCbuntu wizards) and I began questing for a new ppa wizard.

November 9th struck. One more day until unofficial release day. I sent out a team-wide email to make sure everyone was informed that tomorrow was release day. I’d already written the release announcement. The only thing left was to make sure there weren’t any major stoppers.

Spiff had already said if we couldn’t get a ppa guy or figure it out on our own, we were going to go ahead with beta1 regardless. So ppa itself was not, technically speaking, a showstopper.

Needless to say, even without the Linux ppa, there were still several potential stoppers to deal with.

Number one on the list was the lack of the Frodo addon repo. For those not aware, every major release of XBMC comes with its own repository of addons. This is done for a couple reasons, but the primary one is to ensure that we don’t have dozens of outdated addons listed in our repository that don’t actually work. That kind of thing looks good for no one. We had not yet created a new repo, and we certainly hadn’t ported the known working addons over to it. Whether that would stop the beta was unknown.

Additionally, at almost the exact same time as I had sent my email, we had invited wsnipex to join the team and manage our Linux ppa. He’d already had quite a bit of experience in the field and seemed like a natural choice to lead the effort. It was destined to take a certain amount of time before he had access to our launchpad.net unstable ppa account, but that time was absolutely worth it. Whether he would get that time was less certain.

“I’m sure a sonic screwdriver could help you with your buildbots!”

It was beginning to look like whatever beta we put out would be ugly and incomplete, but then, like a gift from a Doctor Who Christmas Special, we were granted a bit more time. Almost in unison, practically all the XBMC platforms suddenly stopped building in our buildbot. November 10th came and went. Throughout the weekend, devs were scrambling to get the our buildbot building again.

Suddenly, we had plenty of time to get Linux going and the addon repo moved over, but on the flip side we had no idea when beta1 was going to come out.

Bringing It All Together

Over the weekend, wsnipex got the keys to the ppa. In very short order, he began performing successful test builds. Piethein (aka Erhnam) used those builds to create XBMCbuntu packages in a split second. The Linux side was looking good.

By Tuesday, Cory had gotten the Frodo addon repo in place.

On Wednesday, Spiff, Cory, Martijn, Olympia, and Montellese all moved as many addons as possible in various categories to the new repo. Only skins were left out, as it was decided each skinner should make the decision whether to move to the new platform or not. Spiff tagged. The Beta splash screen that users see was added to the Beta. And the build errors had all been worked out.

As a great deal of work had been accomplished, many of the devs drifted away. I determined it was time for dinner, but stated generally that I’d really prefer to see the beta released on Thursday instead of the weekend. Martijn noted that since Spiff had tagged beta, there was really no reason not to start building right now. I asked Martijn if he knew how to kick Billy the Buildbot and build the Windows, OSX, and iOS betas. He said he could probably riddle it out. I left him to it.

That evening, after dinner, Cory informed me that the Windows and OSX builds were done. We shifted those over, and I posted links to those builds on our download page.* I decided that it’d be a fun treat for those individuals who happened to stumble upon the downloads page to get an early copy of the beta before everyone else. My plan from that point forward was to post up each install as it came online.

*I also posted a link to our donations page on the downloads page. One of our big goals over the next twelve months is to be able to pay for our own developers conference without relying on the aid of a big corporate entity.** The more XBMC is for the people and financed by the people, the better we believe the whole product will be. That starts with making the ability to donate simpler and more visible. 

**In other words, if and when you donate to XBMC, you are welcome to think of your donation as a contribution to the “Keep XBMC Away from Sony” foundation! (Of course, legally speaking, our Foundation cannot be swayed by big corporate interests, as that would violate our 501(c)3 non-profit status tax protection, but probably just ignore that fact while you donate!) 

With Windows and OSX up, we were still missing Linux, XBMCbuntu, iOS, and AppleTV. It was 11PM on Wednesday night.  I emailed wsnipex and Peithein, asking them if they could get a build off in the next nine hours.

An hour or two later, after going over the announcement with a fine toothed comb and asking several other devs to take a look at it, I emailed a few news outlets who we have pretty good communication with, filled them in on the full story, and asked that they refrain from publishing within the next 12 hours, or until I sent them the go-ahead.

At 2:30AM (aka around 8:30am in Europe), wsnipex returned to the land of the living and began building. An hour later, after one false start, the Linux unstable ppa was ready to go. I updated the download page to reflect another source available.

By 4AM, Piethein had built XBMCbuntu, but a new problem cropped up. Nobody online had the access keys to upload XBMCbuntu.*

*A new board was recently elected for the XBMC Foundation. I was elected secretary, probably because I don’t like to write. ;) I think one of the very first things this board should do is put together a list of all passwords to ensure that we have double and triple backup to access accounts when necessary, regardless of the local time of day. 

Shortly thereafter, Amet appeared. Amet is in charge of uploading iOS/AppleTV builds. Curiously, even though XBMC for iOS has been around for quite a while, this is the first time we’ve had to replace the stable iOS version on Cydia, since the last version, Eden, was the first stable version of XBMC for iOS.

Between Amet, Ned Scott, and myself, we worked out that we could place the beta in a separate repo folder. Doing that allowed people without that new repo installed in Cydia to download the stable XBMC Eden. Further, those excited for the upgraded software would be able to simply add a second url and easily grab access. We had Ned update the wiki explaining all these new options.

That covered Windows, OSX, iOS, and Linux. The only thing missing now was XBMCbuntu. But it was seemingly not possible at that time (or within the deadline set for the press) to get access to that portion of the XBMC Mirror in order to get XBMCbuntu uploaded.  So I made a command decision. We would publish now with a note that XBMCbuntu would be available asap.

This is almost exactly what I looked like!

And so it was, at 7:06 AM on the 15th of November, 2012, Beta1 of XBMC 12.0 was officially released. I emailed various members of the press. I posted to Facebook, Twitter, and Google Plus. I dealt with early comments and cleared some last minute issues and hurdles. And then, exhausted, I fell into my bed. The thrill of a major release can do wonders for keeping a man running well past bedtime, but eventually the adrenaline wears out.

 

Some Final, Personal Thoughts

Man, do I ever love releases. As some of you know, I’ve been unemployed for quite a while now, except for the occasional website contracting job or social media task. Generally, I attempt to combat this by applying for Community Manager or Marketing Manager positions and hoping that employers will be impressed by all the work I do for XBMC and the resulting success we’ve experienced in increased adoption rates, increased social media subscriptions (we now have 30k Facebook followers, up from 6k when I took over, all on a budget of literally zero dollars!), and increased site traffic. But I suppose I’m going to have to work to even harder to make employers pay attention, because so far I don’t appear to have impressed anyone.

Anyway, with this release (and, honestly, the release of Eden before it), I’m beginning to wonder if I somehow missed my calling.

Maybe it’s the team I’m working with. Maybe it’s the multi-platform nature of XBMC. Maybe it’s the way I get to have my fingers in so many pies, working with half a dozen team leaders and devs, the press, people from numerous continents, and users that literally span the globe. I honestly couldn’t say….

All I know is that I wish for-profit companies had some kind “Non-technical Release Manager,” whose job it was to push a software product release out the last few legs of the journey. Because working with this team to coordinate such a monumental release of theoretically boring beta software is better than the best runner’s high, as far as I am concerned.

People always say you should try to turn what you love into a career. I’m fairly sure the world literally doesn’t have a job that lets you do all the things I enjoy doing for XBMC (last spring I even helped re-write the Corporation By-Laws!). But if it did… man, would I ever be jealous of the guy who was more qualified and happened to live geographically closer to the company that was hiring for that position, because he would be one lucky guy!

Some Final Impersonal Thoughts

Whether Beta2 is going to be as tricky a release remains to be seen. The odds seem to be against its being all that unpleasant. Now that we have a ppa maintainer in place and the Frodo repository exists, we should be all set. On the other hand, I’m fairly sure we always feel that way, and some last minute problem always pops up that requires some degree of scrambling. That’s just how life works.

Still, I’m hopeful for beta2, and also somewhat hopeful that we’ll have at least one nice little surprise for everyone that’s still waiting in the wings.

Other Notes

One of the biggest problems we have, building, may become much less problematic in the near future. Cory Fields, often times the primary build manager, is looking to switch from our current buildbot to Jenkins, which should theoretically make the entire process easier and more automatic. If we do end up traveling down that path, I’ll let you all know how it goes.


  • TannerRogalsky

    Thanks for taking the time to put this together. I was very interested to read about  your processes!

  • Jolindse

    I really enjoyed this behind the scene text. Thanks for taking the time to write it!

  • pettergulbra

    Thanks for the behind the scenes. 
    I love the program, and you gays!

    You do so much work that people don`t see,
    and this is an really good way to tell peoplet that!

    • NathanBetzen

      Hmmm.  Typo? :)

  • Mohammad

    Phew, a very long post for someone who doesn’t like to write! Great job man, thoroughly enjoyed reading through the drama and imagining being in your shoes. Grateful to you and the team for such a great release!

  • Vinicius

    I’ve become exhausted just by reading this… Great job!

    Just for your knowledge: ppa is ubuntu-land not linux-land.

    • NathanBetzen

      Yeah, I thought about clarifying that, but then I was like, “I’ve already written like 3000 words!”