As a new year begins, we hope more mobile dev teams will distribute beta builds before releasing to the public. If you're reading this article, chances are you’re looking for the best beta distribution tools and services. This article covers tools for both iOS and Android, but focuses on iOS (and TestFlight alternatives) as it's a pickier platform for beta testing with more complexity.
This article compares each tool regarding multi-platform support, integrations and APIs offered, ease of use, popularity, setup time, miscellaneous pros and cons, and, of course, pricing. I discuss and compare continuous integration and continuous development adoption and integrations, as well as user feedback and usability. I'll also cover ease of use and setup time, analyzing the steps necessary to get everything set up and running. Additionally, I'll mention any well-known customers for each service, as well as any publically available metrics when calculating the popularity factor.
Dealing with UDIDs (Unique Device Identifiers) is one of the things iOS developers struggle with when distributing beta builds. These are part of the provisioning profiles needed to release beta versions. You'll notice that in most tools presented in this article, iOS developers will have to handle the provisioning profiles manually and submit a new build with the newly generated provisioning profile, but some tools can automate this task. Unless stated otherwise, assume that the service does not handle the provisioning profiles for you, so you'll have to deal with them manually.
List of TestFlight alternatives reviewed in this article
Name
Platforms
Pricing
Android, iOS, Others
10 Apps Free
Android, iOS, Others
2 Apps Free
Android, iOS
Free
Android, iOS
Free
Android, iOS, Others
2 Apps Free
Android, iOS, Others
1 Concurrent Build Free
Android, iOS, Others
25 Devices Free
Android, iOS
Free For Indie Developers
Android, iOS
3 Devices Free
Android, iOS, Others
2 Projects Free
Top 11 iOS Beta Testing Tools comparing @testflightapp #hockeyapp @buddybuild @crashlytics @testfairy and more https://t.co/eQMYfaE7hp
— Rollout.io (@RolloutIO) January 31, 2017
TestFlight
Popularity: If you're into the Apple development world, you have probably heard of TestFlight. There are many opinions for and against the use of TestFlight since Apple acquired it back in early 2014, but during its first 3 years of existence before its acquisition, it had already gained much popularity and adoption.
Platforms: Since its acquisition, Apple stopped supporting Android distributions, and it now allows distribution to Apple devices only - iOS, watchOS, and tvOS.
API & Integrations: Fastlane, more specifically it's `pilot` lane, will be your best friend when delivering your app to TestFlight's beta testers. With a single command line, you're able to get your app from development to your beta testers' hands.
Ease Of Use: If you're not using an automation tool like Fastlane in order to automate your workspace, it's still pretty straightforward to get your app into beta testing. The whole process is done through the iTunes Connect portal, and you can add either Internal or External Testers. Internal testers are "part" of your development team, thus only the ones that are currently assigned as developers or have developer roles in your iTunes Connect team (like admins). The app distribution sent to internal testers are the fastest since they don't require a Beta App Review nor comply with the App Store Review Guidelines, like the distribution sent to external testers.
On the other hand, to invite external testers, who is anyone that's not in your development team, you only need their email address and they're good to go once they've accepted the invitation. Only TestFlight has this Internal and External Tester concept.
TestFlight's advantage is that it doesn't require the developer to handle the provisioning profiles for new devices manually, so that improves its ease of use considerably.
Setup Time & Effort: The setup time is mostly creating the app in the iTunes Connect and submitting the app via Xcode, which could take up to a few minutes to get your app approved through Apple's automated app revision system. You need to explicitly declare some information about the test build you're submitting, like a description and what needs to be tested, so that might take a little longer than if it didn't require that info.
Misc Pros: Every tester needs to download Apple's official TestFlight app in the App Store in order to be able to download and test app distributions. Another great feature is that you can assign notes and details to the app being distributed so the testers can see what needs to be tested.
Basic information about crashes and sessions are stored in TestFlight Data so the developers may track what's going on with each of the versions that are being tested. Extra feedback may be sent to the developers via email (within TestFlight app) alongside a log with information of the current device.
Misc Cons: What bothers the developers the most is the Beta App Review that is required in order to invite external testers (most times the most important testers are outside of our development team). So that, and the fact that it doesn't support non-Apple platforms, are the reasons why many people complain about TestFlight. Other than that, it's a great tool, it was already one of the most popular (if not the most) on the market before Apple bought it, and it definitely became even more popular after its acquisition.
Pricing: Apple's official solution to beta distribution is completely free to use.
Other: A quick video tutorial is offered by Apple here. Alternatively, a full tutorial is available on Ray Wenderlich website.
TestFairy
**
**
TestFairy is one of my recommended TestFlight alternatives. It works for iOS and Android and dozens of other platforms, and provides more than just app distribution. TestFairy provides videos showing what exactly happened on your app before something went wrong. The service provides users with an option to shake their device and send feedback back to the developer. This feedback, can be automatically be pushed to Slack or JIRA, with a video that shows what exactly was done with the user. Integrations & APIs: TestFairy's list of integrations is really great! Bug tracking integrations include JIRA, GitHub, Trello, Bugzilla, Asana, Pivotal, YouTrack as well as Slack and HipChat. Continuous Integration: TestFairy has an open API for uploads and plugins for popular CI tools like Jenkins, Travis-CI, Circle-CI, Bitrise, Fastlane, Bamboo. Platforms: TestFairy is available for native iOS, Android, Xamarin, React Native, Adobe Air, PhoneGap, Cordova, ionic, Appcelerator Titanium, and Unity. Popularity: TestFairy's quite popular with the enterprise crowd. Clients include Samsung, Groupon, Adobe, Johnson & Johnson, HP, Century Link, Sears, McKinsey. Enterprise Hosting: TestFairy's enterprise edition is available in Private Cloud or On Premise and has an option for white labeling. TestFairy for startups is available in public cloud only. Single Sign On: TestFairy supports SAML and LDAP, and is compatible with Okta, OneLogin, Ping Identity, Google Apps, Oracle and IBM's SSO solutions. Setup Time & Effort And Ease Of Use: You don’t even have to add an SDK to your project if you’re only looking for the app distribution feature, so it’s one of the easiest tools to setup and start using. As a plus, you also get in-app bug reporting. Pricing: TestFairy has a basic tier that is free, and offers paid packages for startup and enterprise companies.
HockeyApp
Popularity And Misc Pros: More complete than TestFlight, and definitely among the most popular beta distribution services out there, Microsoft's solution to beta distributions also offers user feedback right from the app, insights of how users are using the app, sophisticated team management features, and is well-suited for large companies that develop many different apps.
Platforms: It also supports a greater range of platforms, from Android, iOS, macOS and Windows, to Cordova, React Native, Unity, and Xamarin.
Ease Of Use And Setup Time & Effort: The setup is pretty much the same as Fabric, which I'll be covering later on. There are numerous ways to add the HockeyApp SDK to your project, including CocoaPods, Carthage, and manually. Uploading a new build may be done via the web dashboard or via HockeyApp for Mac, and is really straightforward.
Integrations & APIs: It features a huge range of integrations and bug trackers, and provide handy tutorials for the most popular ones like JIRA, Slack, GitHub, Status Board iPad app, Microsoft's Visual Studio Team Services, and also webhooks. Additionally, HockeyApp also supports various build servers & tools: Fastlane, Jenkins, Travis CI, and Visual Studio Team Services.
Pricing: All this is available in a free tier for up to 2 apps, and the other plans can be found here. For those looking to use more than 2 apps and wouldn't like to pay more for it, HockeyKit open source might be an alternative.
HockeyKit
Platforms: From HockeyKit GitHub Wiki, HockeyKit is a framework to easily distribute applications with maximum ease of use for end-user and great flexibility for developers, which currently supports iOS and Android platforms.
Integrations & APIs: It's an open source framework that preceded HockeyApp. It's an alternative for those who have their own servers and would like to host their own beta distribution system, but it hasn't been updated for over 2 years and is provided as is. For further support and features, they recommend using HockeyApp, as it's the business that proceeded this product.
Ease Of Use & Setup Time & Effort: Since it's an open source framework that must be fully installed on your own server and is provided as is, it isn't easy to use at all, or at least takes a certain time to have everything set up. It has a good documentation, though, and since it's a self-hosted solution, you can integrate pretty much anything you need or want.
Popularity & Pricing: It isn't as popular as HockeyApp, but then again, it's completely free to use, which makes it the best solution if your company doesn't want to rely on 3rd party software to deliver beta builds.
Crashlytics Beta (Fabric)
Ah... Finally, Fabric. By far my personal most favorite suite of mobile developer tools. Owned by Twitter, this suite of tools include a total of 14 SDKs (and counting!), into a single, modular SDK, allowing developers to choose which tools they need for each of their projects, and easily integrate them into each of them. Additionally, Twitter crosses Fabric's Answers Analytics information with Twitter's analytics, so you get absolutely amazing audience information without paying any extra for that - a unique feature that only Twitter is capable of offering to its developers - and that's one of the reasons that makes Fabric my personal winner for best free tool!
Setup Time & Effort And Ease Of Use: Focusing on Beta (which's why you're here), the setup works just like HockeyApp, as I have mentioned before, so it's pretty straightforward, and everything can be done either via your favorite dependency manager.
Integrations & APIs: I strongly encourage using Crashlytics alongside Answers and Beta, and, of course, why not integrate beta to Fastlane? With the press of a button, your builds will be available to all of your beta testers, simple as that. And since Fastlane development is now operating under Twitter's HQ, they put up some serious effort into making it the most seamless beta distribution tool on the market.
You can also add Crashlytics to your team's Slack channel. It's not really the same as integrating with Beta, but whenever you have important errors and crashes (including in your beta distributions), you'll have them notified in your Slack (or in your email too, if you're old-fashioned).
Platforms: Beta supports Android and iOS.
Misc Pros: Fabric also provides an amazing real-time heartbeat-like chart where you can see how your production or beta app is doing.
Popularity: If you're still not convinced, Fabric was serving over 1 billion active devices after only 9 months of its launch, and by the end of 2016, Fabric was serving over 2 billion active devices (virtually all the devices on the Earth), while landing top spots for app analytics, stability, and monetization of 2016.
Pricing: Lastly, best of all: 100% free.
Update: Google bought Fabric in January. It's hard to tell what that means right now, but Google's latest acquisitions have had the most variable results, like shutdowns and service restrictions. We can never tell, but I don't think Fabric will have a sad ending like that. All we hope is that Google doesn't let us down with Fabric's 2.5 billion active mobile devices!
Applivery
Another great solution that stands out is Applivery, mainly because of its customizability when distributing your builds, which makes it a great solution if for enterprise distributions (those that are not meant to be distributed in the public app stores).
Platforms: It supports both native and hybrid apps on both Android and iOS platforms, under any programming language supported by those platforms.
Integrations & APIs: It also provides one of the most integrated solutions, offering plug-and-play integrations with Slack, JIRA and Google Analytics. It also allows uploading apps automatically, directly from your build server, being prepared to use Bitrise, Jenkins, Travis-CI or any other continuous integration, deployment and delivery platforms. To top on that, it even provides a Fastlane plugin that will deploy everything with the touch of a button, just like its competitors. And if you want even more direct access to its features and functionalities, it offers a huge REST API covering everything.
Setup Time & Effort And Ease Of Use: If you're not used to Carthage, you'll find yourself starting to dig into it, unless you'd like to install it manually, since CocoaPods isn't an option with Applivery. Sending feedback or bug report is as easy as taking a screenshot when using Applivery, so it adds on great usability.
Popularity: Since it's a recently founded startup in Spain, it isn't as popular as the previously mentioned solutions, but it's definitely a strong competition to the market, due to its enterprise-focused solutions.
Pricing: Due to its B2B nature, it's not completely free for all of their features, but the free version currently includes 2 apps, 10 builds/app, 5 distribution sites, 2 different teams and 500 installs. For more information on the other plans, see their pricing table.
buddybuild
If you have some budget to invest on your project, my personal favorite is definitely buddybuild. With an outstanding range of integrations, auto provisioning profiles management, huge platforms support, and a full continuous integration and deployment support, it just makes sense why all those top tier companies are using it.
Popularity: As stated in their website: "Some of the world's most respected brands trust buddybuild" and I wouldn't change a single word. Slack, Arc'teryx, Firefox, Meetup, NBC and Monzo are some of the notorious names, which makes it pretty impressive.
Integrations & APIs: Just like TestFairy, buddybuild's integrations cover pretty much everything you can imagine, from building phase, to deployment and user feedback. Integrations include any git based source code repository, iTunes Connect and Google Play (to automatically deploy TestFlight betas, or put your app into production into the app stores), and most communication and bug reporting tools like Slack, HipChat, Trello, JIRA, GitHub, Pivotal, and more.
Platforms: Android, iOS, and surprisingly tvOS and watchOS are part of the platforms that buddybuild supports. If you're developing hybrid solutions, they also support any hybrid development framework like Ionic, React Native, PhoneGap, and more.
Setup Time & Effort: Feel free to use any dependency manager you're comfortable with, like CocoaPods, Carthage, npm or Maven, so all dependencies are automatically detected and installed. It doesn't require any configuration setup, scripts or files upon setup, so everything is as smooth as possible when getting everything ready to go.
Ease Of Use: When it comes to ease of use, definitely this is one of the most outstanding solutions on the market. Again, with their words: "buddybuild ties together continuous integration, continuous delivery and an iterative feedback solution into a single, seamless platform" and works absolutely beautifully. With every `git push` to your git server, buddybuild will trigger a build of your app, run any unit or UI tests (even on physical devices, if needed), and prepare your app to be deployed to App Store or Google Play. And remember the UDID handling mentioned in the beginning of this article? Buddybuild handles everything for you, so you don't have to worry about it.
When it comes to the feedback phase, your beta testers can report bugs and give feedback simply by pressing the screenshot shortcut on the device, and even highlight an area in the screenshot if he wants to. Additionally, with every bug or crash report, an instant replay is available for your to analyze, giving you an image on how to reproduce that specific bug.
Misc Pros: What is even cooler about them is that they can send you a quick demo so you can see how easy it is to get everything working. They email you a beta version of a 2048 demo app so you can play with it, find out bugs or crashes and report feedbacks. The only bad thing about it is that I personally spent 25 minutes playing 2048 when I first received the demo app. ¯_(ツ)_/¯
Pricing: Their free tier offers 1 concurrent build, auto provisioning of 5 new devices, 10 screenshots/month and 5 instant replays/month. Higher plans can be found in their pricing page.
Other Companies Worth Mentioning
AppBlade
Highlight: AppBlade is a Mobile Device Management tool, thus it's a beta distribution tool with a focus on security. You can remotely install, update and uninstall apps into your devices, as well as secure them, add a layer of encryption, and wipe info from the devices, everything remotely via its dashboard.
Platforms: AppBlade is available for the iOS, macOS, Android, & BlackBerry.
Pricing: Free for the first 25 devices, and as cheap as $1/device/month afterward.
DeployGate and Installr
Highlight: Both of these services register your testing devices connecting to Apple Developer Portal, so they auto-manage your UDID, handle your provisioning profiles and update your IPA automatically. If your business constantly uses new testing devices or testers, this is definitely the feature you were looking for.
Pricing: DeployGate's has fair pricing for indie developers (from $0 to $15/month), but quite aggressive pricing for enterprises, but that's up to each business to decide. You can find their pricing table here.
Installr on the other hand has a really limited free tier, and charges "per app".
Ubertesters
Highlight: In case you'd like your app tested but don't have a QA team, Ubertesters offers exactly what it sounds like: a platform with distributed independent testers to test your app. The testers are divided in 3 different categories: functional testing, usability testing, and localization testing, so you can hire the testers that will better test your app for your needs.
Alongside with the crowd testing platform, it also offers a QA management tool so you can monitor and manage all processes and your own teams in real time.
Pricing: The on-demand crowd testers service starts at $25/hour. Ubertesters do offer a free tier of services, but their full pricing table can be found here.
Comparison Chart
================================================================================================================================================
Conclusion
Each of the 11 tools analyzed here was worth mentioning since each has its own niche, specialty, and highlights, so, depending on your application, you might want to use different beta distribution tools.
I highly recommend start getting used with Fastlane and its amazing toolkit. It integrates with most of the services listed in this article, and that's not just coincidence.
If your app is restricted to the App Store (and you don't have any plans on developing an Android version in the future), my choice would be Apple's TestFlight due to the ease of setup, auto provisioning profile management, and the same build that you use on it can go live on the App Store with the press of a button.
Considering all the pros and cons, it wouldn't be fair to choose once single winner due to the difference in pricing.
Let 2017 be the year of continuous integration, delivery, and deployment, better app testing, and more integrated workflows. Let's get together with the QA team - they are our friends!
[ulp id='a4AgL3TDrA8iWV9n']