My Ubuntu for mobile devices post mortem analysis

Now that Ubuntu phones and tablets are gone, I would like to offer my thoughts on why I personally think the project failed and what one may learn from it.

To recapitulate my involvement in the project: I had been using Ubuntu Touch on a Nexus 7 on an on-and-off-basis between its announcement in 2013 and December 2014, started working on Click apps in December 2014, started writing the 15-part “Hacking Ubuntu Touch” blog post series about system internals in January 2015, became an Ubuntu Phone Insider, got a Meizu MX4 from Canonical, organized and sponsored the UbuContest app development contest, worked on bug reports and apps until about April 2016, and then sold off/converted all my remaining devices in mid-2016. So I think I can offer some thoughts about the project, its challenges and where we could have done better.

Please note that this post does not apply to the UBPorts project, which continues to work on the phone operating system, Unity 8 and other components.

1. It didn’t target a profitable niche.

Ubuntu for PCs, laptops and servers had it relatively easy. Pretty much all of these devices allow you to install any operating system which knows how to use the hardware, and when Ubuntu popped up in 2004, its biggest opponent (Microsoft) was very vulnerable. Windows had a bad reputation, was quite expensive and a resource hog, so Ubuntu only had to be less annoying, cheaper, easy to install, and run better on older computers. And that’s exactly what it did. Windows continues to have a bad reputation, is now even spying on its users, and is still quite expensive. So the Ubuntu Desktop didn’t and doesn’t have to do many things right to maintain and increase its user base.

On the server side, Windows, Red Hat and especially SUSE were perceived as being too conservative, too slow-moving, and again too expensive. A Red Hat Enterprise subscription goes for several hundred dollars a year, and that subscription doesn’t even necessarily contain actual support by humans. A fast-moving, less costly alternative with some industry support and a huge number of packages in its repositories had to be interesting for many people, especially in the cloud. Ubuntu becoming the reference operating system for OpenStack helped a lot as well.

But on the mobile devices side, everything is different. You can’t just flash any generic operating system on your phone or tablet. Every device basically comes with a custom, specially tailored build of Android. When Ubuntu announced to step into the mobile market in 2013, neither Android nor iOS were vulnerable, on the contrary. Android had pretty much already pushed every other platform except iOS out of the market. People crying for a third alternative mostly didn’t do so because Android or iOS had a bad reputation, or were too limited, or gave people a bad user experience, but because they (rightfully) feared a Google monopoly. So attacking Android and iOS was not as easy as attacking Microsoft and Red Hat on PCs and servers had been.

I remember someone from Canonical saying that the project needed to capture about one percent of the mobile market to sustain itself. At that time this translated into selling about eleven million Ubuntu phones and a couple million tablets per year. If you manage to make at least one euro/dollar for software and services per device, you can easily pay more than a hundred developers, and that’s a lot if you use them right. Jolla, the company behind Sailfish OS, had about 120 employees at some point I think, but they also needed marketing and support departments, which Canonical already had. But selling eleven million phones and a couple million tablets per year was a very ambitious target, considering that the number of Ubuntu Desktop users was estimated to be somewhere around twenty to thirty million.

• Possibility #1 to get to one percent: Be so much better than the competition that you become mainstream and don’t even have to worry about the one percent. I think we all knew this wasn’t going to happen, especially when it became clear that all the important services (WhatsApp, Google, Twitter, Instagram etc.) wouldn’t even allow clones of their apps to run on Ubuntu devices. If Canonical hadn’t built their own Telegram client, there wouldn’t even have been an Instant Messenger when the first commercial Ubuntu phones came to market. And this in the year 2015, when everybody used Instant Messengers all the time. Nobody is going pay the same amount of money for an Ubuntu phone if it can’t do the same things as the same model with Android, even if it’s marketed as a “developer device”.
• Possibility #2 to get to one percent: Cater to a niche with deep pockets. Canonical focused too much on the “Convergence” niche, which not enough people cared about, while they pretty much ignored all the hackers, tinkerers and people who were fed up with Google, Microsoft and the NSA snooping on them. Not many people are ready to pay a premium for a phone which may turn into a slow laptop when connected to an external display, but enough people are ready to pay the premium for a Blackphone.

2. The user experience was bad and priorities skewed.

I want to be honest here: After the first couple of over-the-air (OTA) updates had shipped, I asked myself “Do bq and Meizu, and especially their customers, really put up with this?”. The phones were slow and had to be rebooted on a regular basis. The Meizu MX4 overheated. The battery indicator tended to show bogus data. Mobile data was unreliable, (national) roaming often didn’t work at all. The location service was very unreliable. The phone didn’t always ring when called, or you couldn’t make an outgoing call because the UI hid the buttons. The alarm didn’t work reliably. Bluetooth only supported audio devices, and later input devices, but not even basic file transfer. WiFi would not connect to WPA Enterprise networks until OTA-5. I think at one point the music player even started deleting files while indexing them. Et cetera.

The list of things you expected to be working, and then didn’t, was very long. But even worse, bugs would more than once come back two OTAs later as a regression. During the lifetime of the phone/tablet project, the number of bug reports on Launchpad skyrocketed like I’d never seen before.

Eradicating all those bugs was not exactly a top priority, the developers spent most of their time working on support for more hardware (Meizu Pro 5, bq Aquaris 10) and working on Convergence. Up until the very end of the project, most users I spoke to on a regular basis weren’t happy with their device. Only those relying on the most basic functions, like my father, who doesn’t even have mobile data and makes a call every two days, were quite happy because the device could run for days. Buying a 150 € smartphone and then not using any of the features that make it “smart” doesn’t make a lot of sense, though.

I understand there weren’t enough developers to fix everything at once, but instead of deciding to either make a good phone OR a good tablet with Convergence, we had devices which couldn’t really do anything right. The whole project also always always had this “these are developer devices, it’s not important to do it fast, we will win in the long run” air around it – until the management quite obviously realised that this was all way too expensive and too much time had already been lost. That’s when they cut their losses internally, moved all key people over to Snappy around October 2016, let the phones and tablets die a silent death, and didn’t tell the public until months later.

Oh, and I think the designers held onto the “Scopes” idea way too long. Especially when noone could really find out how to use Scopes on a desktop.

3. The devices were hard to get and didn’t deliver.

I think we can all agree that it was way too hard to actually get a device. I bought my first Nexus 7 at a store and my Nexus 4 on eBay, but when the project actually picked up serious momentum, those devices were already old, harder to get and soon there weren’t any offcial image builds for them anymore. The bq devices were at least sold in most of Europe, but were “out of stock” more often than not. Getting an MX4 was a pain for everyone except us Ubuntu Phone Insiders. If people in the United States of America even got hold of a device, it would most often not connect to the mobile networks at full speeds.

For most of 2015 and 2016, if app developers wanted to get an officially supported device to test their apps on, I didn’t even know which one to recommend.

On the other side, the device most people expected – the extremely high-performance Ubuntu Edge – was not the device people got. The bq devices were cheap, had small internal storage, and only 3G. The MX4 was fast and had a big screen and 4G, but nothing else, not even an SD card slot. The HDMI output needed for Convergence was missing on all official phones, and Miracast/Aethercast was not an equivalent solution. Lots of people thought Ubuntu would unlock the full potential of their hardware, e.g. the FM radio on the Aquaris E4.5/E5, but that was not even on the feature list and without the Android sources for the device drivers it was nearly impossible for the community to add that feature.

Many people also expected their Ubuntu phones to be inherently more secure than Android because of the open source software and frequent updates. That obviously wasn’t true, the Android driver bits and the mobile baseband were still proprietary and insecure and had full access to the hardware. But that’s not something many people could grasp.

4. Communication and marketing were rather chaotic and sometimes misleading.

I spent a huge amount of time every day trying to keep up with development, but most of the time even I didn’t know what would come next or what would end up in the next OTA. Mailing lists, IRC, Telegram channels, Launchpad, the official web sites, private conversations between developers, sprints, Ubuntu Online Summit – there was just too much. And that doesn’t even include all the non-public conversations going on at Canonical when they had to keep a secret to ensure maximum news coverage when they unveiled it.

With many of Canonical’s employees working from home or at least in different time zones, for me things often became even worse. I remember when I tried to help with the “When I press the power button, the phone only wakes up a second later” and the “bogus battery indicator” bugs. The only constant was Launchpad, and people were expected to automatically make the most of it. But often it’s much more efficient to talk to a person for a minute before you can actually contribute anything of value to a bug report, or just to decide how to approach the problem.

The person working on the kernel source of the device might have been somewhere in Asia. The person doing all the Q&A might have been somewhere in the USA. I was in Europe. Our work shifts didn’t really overlap. So on some days I would find myself talking to the guy in Asia at 8 AM before he left work, and then talk to the person in the USA in the afternoon or at night when they had just started.

I have to say I learned a lot from the marketing department, especially when it comes to “expectations vs. reality”. For example lots of people just assumed that the Aquaris E4.5/E5 and the MX4 would also get the Convergence feature with a later OTA update, but that was something neither the manufacturers nor Canonical had advertised when selling the devices. Up until the project was cancelled, most people silently assumed to be able to run the same applications as on the desktop (Firefox, SIP clients etc.) and manage the phone with apt-get, and that’s where the whole marketing became simply misleading. There was too much emphasis on “It’s all the same Ubuntu” when it really wasn’t. I cannot remember how often I had to explain to random people on various support channels that Firefox would not run and using apt-get would break things. Often people were completely surprised when they found out that Ubuntu for mobile devices was so much different.

5. There was too much focus on technical features the users and app developers didn’t care about.

I feel like the announcement of a new and independent mobile operating system platform was a good reason for the architects to say: “Yeah, let’s do this, but let’s do it The Right Way™ and be better than all the others”. Ubuntu wouldn’t just have a graphical user interface, it would have one that could work on all devices and adapt to all form factors. It wouldn’t just isolate applications against each other the way the Linux kernel or Android did, it would have full-blown confinement which also protected your data and privacy. It would magically prevent apps from draining the battery. And so on. Whatever the others did on the technical side, Ubuntu would do it better and in a more elegant way.

Not all of this made sense to me. Unity 8 was needed because Unity 7 depends on Compiz and is not well-suited for working on a variety of form factors with rotating displays and so on. But the only job Mir had was to replace X.Org and SurfaceFlinger, so Unity 8 could use a single API on PCs and mobile devices. I’m not an expert on graphics technologies and APIs, but at least from a “we are quite short on manpower” standpoint it feels like coming up with a whole new display server which noone else wants to use and doesn’t add much over the existing alternatives should have been avoided at all costs. Especially when the user never sees the difference. Ubuntu Touch had been happily using Android’s SurfaceFlinger until the end of 2013.

The same is IMO true for the confinement and lifecycle model. If your design is complex for the sake of maybe saving a tiny bit of battery power, and this complexity leads to a lot of additional work for the implementation of system services, but those services never get implemented because your team is too small, then your users and app developers most likely won’t applaud you because their devices last a bit longer, but they will complain endlessly about the missing stuff. That’s why the “Complete the high priority background service implementations” bug report on Launchpad rose to a heat of 240 – three years had passed since the announcement of the project, but pretty much nothing had improved in this regard.

Another good example is the planned messaging framework. You would only have one system application for all your messaging needs, whether it’s Jabber/XMPP or SMS or Telegram or WhatsApp, and third party sources would be able to ship plugins for their services. This framework was one of the major reasons for denying apps the right to run in the background. So you couldn’t make a simple standalone XMPP client that would continue to receive messages in the background, but the messaging framework you were supposed to plug into was delayed until it never actually shipped. Even the Telegram client couldn’t run in the background, it was only able to show popup notifications because Canonical convinced the Telegram developers to change their server code (!) to support the Ubuntu Push Notification service.

Some key developers over at Canonical really thought Ubuntu was so important that all service providers would change their server code to use the Ubuntu Push Notification service, solving the problem. Noone except Telegram ever even gave this a thought.

(By the way, I still believe the lifecycle model wouldn’t have resulted in a noticeable amount of battery savings. Thinking it through, it would have always spent the same number or most often more CPU cycles for the same operation.)

6. The life of an app developer was too hard.

The value of a mobile operating system platform is not in the base operating system nowadays. It’s in the ecosystem. And this is where Ubuntu actually crashed the hardest.

Ubuntu for mobile devices was fundamentally incomaptible with every runtime environment which had existed before. It couldn’t run Android, Windows, X11 or iOS apps. You couldn’t just cross-compile Android, Windows, X11 or iOS apps. The graphics system, system services, confinement, set of base libraries, it all was different. It was even completely different from an Ubuntu Desktop. You can tell me “It’s all the same Ubuntu” all day long, but if I can’t even test my apps on the desktop because it doesn’t even run Mir, then it just isn’t the same Ubuntu and I have to develop for two different platforms.

Canonical went and built a whole SDK, an Integrated Development Environment based on Qt Creator, plus a cross-compiling environment, plus a whole new set of Ubuntu QML components. I really don’t mean to offend anyone here, but besides not reusing existing code, the way this was done was also extremely confusing and frustrating for app developers. Things would not work and change and break all the time. Sometimes the SDK was broken for weeks. Versioning schemes were introduced and your app would break nonetheless.

At some point I had to rebuild and update my glmark 2 app in the store because an OTA came with updated Mir client libraries, while the OS still advertised the same level of compatibility as before. It then became clear that the versioning scheme just guaranteed that the officially supported ways of writing an app were still working, but the officially supported ways were just QML and HTML5. glmark2 talked straight to Mir, as many others (e.g. games using SDL). Apps in the store could just stop working if not continually checked and updated after every OTA. You can still run old Android apps on a recent Android phone, but you had to fear your Click app from last year would no longer work after the next OTA if you didn’t take care of it for too long. I remember a vivid discussion on IRC in late 2015 during which several Canonical developers where baffled to find out about this, asking the SDK team how they expected app developers to work in such an environment.

I started out as an app developer. Whenever I wanted to do something, I pretty much had to start from scratch. Build a GUI? The only supported way was QML with the Ubuntu QML components, and QML is not exactly an established ecosystem with lots of existing code and good tooling. Just use one of the existing UI libraries? They all expect X11 or maybe Wayland, and it took a very long time until SDL and such had backends for Mir. Talk to hardware stuff and system services? Because of confinement I had to talk to special Ubuntu services via D-Bus, most of the “standard” stuff like NetworkManager could not be talked to from within confinement. Download something in the background? Talk to the special Ubuntu download manager service. Get notified of stuff happening outside the phone? Only if you integrate everything with the Ubuntu Push Notification service.

That’s why I actually started to work on the base system. I wanted to build WiFi and Bluetooth scanners back in January 2015, but all the necessary APIs and system services didn’t exist yet. Many of these missing APIs and system services would never appear.

All this made the platform extremely unattractive to most third-party developers. They didn’t see how their investment into building another version of their app from scratch, especially when looking at the very small potential userbase, could be worth it. I can’t think of a single app in the Click Store which was uploaded by its “original” developer. Even Telegram was developed by Canonical themselves.

So many of us went and built cheap web apps or clones of existing apps. And immediately ran into the problem that a lot of apps rely on some kind of non-free online service with very unfavourable terms of service. I personally built BD Navigator, a clone of the Deutsche Bahn Navigator. I reverse engineered their client-server protocol down to the point where I would have been able to replicate pretty much everything except buying an actual train ticket, but they built a small piece of encryption into it, and using stolen encryption keys is illegal in Germany. I asked Deutsche Bahn if they would allow me to do it, they said no. In the end the whole app was degraded to just being a glorified web container with bookmarks for their mobile web pages.

The exact same story is true for WhatsApp, Twitter, Instragram, Google Plus, Google Drive etc. We could have replicated a lot of stuff, but the service providers didn’t allow us to. WhatsApp allegedly wanted a seven-figure sum to just allow access to the API, not for developing a working client app. Instagram locked down their API so much that the built-in Instagram Scope had to be removed. Google doesn’t even have a public API for many services.

(Also a small number of open source nerds would have never been able to keep clones of all popular apps up to date. It just doesn’t scale.)

7. It wasn’t as open and community-driven as intended.

Now I understand that this point might be the most controversial one, and if you don’t agree, please keep in mind that this is what it felt like to me. I might be in the minority. I have no idea.

Ubuntu for mobile devices was supposed to be as open as the “normal” Ubuntu, but it wasn’t.

• The source code for the stuff we developed was all there, somewhere, spread over an unknown number of Launchpad projects.
• The kernel sources were on GitHub and often outdated.
• The code for all the proprietary Android drivers and other bits was only available to some employees at Canonical.
• Canonical and their commercial partners had a whole private Launchpad area with private bug reports. Quite often links on public bug reports would point to private reports, so you only had like half of the information.
• Most bits of information the community had about upcoming devices was obtained by searching through vast amounts of unintentionally leaked data, mostly from paste.ubuntu.com.
• When learning about an upcoming feature, we would often find out that the corresponding Launchpad project had been started weeks or months before under a code name, or that Canonical developers had been working on them in private repositories for months. This was for example true for Aethercast.
• As someone not working for Canonical it was hard to get an overview over what was being done, what was planned, and where you could join in and help out.
• If you found something where you could help out, it was very hard to keep up with the Canonical developers. Their work day had at least eight hours, but you didn’t have eight hours of free time, and your free time often didn’t overlap with their work hours.
• I’ve never felt that the wishes of the users and the wider community had much of an impact on what feature was developed next or what went into the next OTA. In many cases the Launchpad bug reports and feature requests with the most “heat” would linger the longest.

FAQ

Here are some of the questions people are asking me sometimes.

How many devices did you buy for development?

I think I bought two new Nexus 7, two used Nexus 4, three new bq Aquaris E4.5 and two cheap chinese Mediatek phones (used for reverse engineering) specifically for working on Ubuntu. I also got an MX4 from Canonical. I think I’ve spent something over a thousand euros on seven phones and two tablets.

Did you ever estimate how much time you’ve invested?

Yes. After doing all the math I think I’ve invested six man-months over a period of 18 months, or the same amount of work I would have done if Canonical had hired me to work 30% part-time for them.

Do you regret investing so much time and effort?

No.

When did you first start to doubt it would work out?

If I remember correctly, it was around Christmas 2015. A big part of the momentum seemed to have gone away, it had become clear that we would never have full WhatsApp, Twitter etc. apps, and you could start to see that not much of “real” importance for the many phone users was going forward anymore. Convergence for the tablet was being worked on, but not many got a bq Aquaris M10 tablet.

Remember that I had only started working on the base operating system because I couldn’t build half of the apps I wanted to build. After a year of work, not a single of the APIs and system services I needed to do so had been finished, and we were still arguing with some of the system architects if the system should even have all the stuff we needed. When your app developers tell you they need something to build cool apps, just give them something you as an architect can live with for the foreseeable future. It doesn’t have to be perfect, but you need your app developers more than they need you.

You left in mid-2016, long before the project was shut down. Why?

On one hand I had become less interested in software development in general. Nowadays I spend most of my free time traveling the world, taking photographs and creating bad card games, bad comics and bad games.

On the other hand I no longer felt working on the project made me happy. Sometimes I would think “I’m not doing enough, this is my fault” after a sitting over a device for eight hours straight. That’s not how something you do for fun in your free time should feel, I thought.

(Featured image copyright Maurizio Pesce, this picture is licensed under the Creative Commons Attribution 2.0 Generic license)