Mobile DevOps for Enterprise that Just Works
Mobile DevOps for Enterprise that Just Works
CI/CD tools for the problem child
Mobile has always been the DevOps problem child.
You can’t just deploy—teams have to run the gauntlet of app review with each release, and you’re in trouble if you didn’t feature-flag the latest regression.
On iOS, you can’t even just build—in his quest to capture every dollar that’s not stapled down, Tim Cook dictates that we must build on Mac hardware.
The result? Infra teams frequently abdicate mobile DevOps, and mobile engineers are left to make it work, often within the same environment as everyone else.
Today, I usually roll Fastlane scripts on a self-hosted Mac Mini using GitHub Actions. We looked at paying for a MacOS cloud runner but, after seeing the number of zeroes in the price, our ops team didn’t speak to me for a week.
It’s been a hot minute since I evaluated the CI/CD space in detail—the last time was on a banking client pre-pandemic. That’s how I found out the hard way that banks really don’t like IP leaving their premises. In the end, we ended up throwing away my research and hosting an on-prem Xcode Server instance, glued together with bash scripts and cron jobs 🤦♂️.
Sponsored content notice
This article is sponsored content, but views expressed are my own—I am providing an honest assessment as I compare mobile DevOps solutions.
The space is more vibrant these days. I’ve been looking into the new kid on the block, Appcircle. Founded in 2019, it claims to be the Enterprise-Grade Fully Automated Mobile DevOps Platform. I want to find out whether it’s worth migrating away from GitHub Actions for my projects.
For my evaluation, I’ll use Bev, my trusty open-sauce project, as the bachelorette looking for her continuously-integrating prince charming (this metaphor needs work). I’ll set up a CI/CD pipeline on both GitHub Actions and Appcircle from scratch and compare how they perform on a few key metrics:
Ease of setup
Infrastructure management
Build speed
Permissions management
Unfair advantages
Ease of setup
First things first.
While much of CI/CD is set-and-forget (at least, until the new version of Android Studio or Xcode breaks everything again), the ease of setup and ease of making changes are vital when considering the toolset you want to be stuck with.
The time to set up the first build is a critical early consideration for any ops tool. Do you need a roomful of staff engineers working round-the-clock to get that first ✅ or can the intern get it done on their lunch break?
Setting up GitHub Actions
In a classic case of Microsoft bundling, Satya very cheekily cross-sells GitHub Actions in the very UI of your repo.
Always happy to take a shortcut, I tap the iOS lane to see where we end up. It conveniently brings us to a… less than friendly starter workflow.
The flow contains raw xcodebuild commands, which aren’t exactly clear.
steps:
- name: Build
env:
scheme: ${{ 'default' }}
platform: ${{ 'iOS Simulator' }}
run: |
# xcrun xctrace returns via stderr, not the expected stdout (see https://developer.apple.com/forums/thread/663959)
device=`xcrun xctrace list devices 2>&1 | grep -oE 'iPhone.*?[^\(]+' | head -1 | awk '{$1=$1;print}' | sed -e "s/ Simulator$//"`
if [ $scheme = default ]; then scheme=$(cat default); fi
if [ "`ls -A | grep -i \\.xcworkspace\$`" ]; then filetype_parameter="workspace" && file_to_build="`ls -A | grep -i \\.xcworkspace\$`"; else filetype_parameter="project" && file_to_build="`ls -A | grep -i \\.xcodeproj\$`"; fi
file_to_build=`echo $file_to_build | awk '{$1=$1;print}'`
xcodebuild build-for-testing -scheme "$scheme" -"$filetype_parameter" "$file_to_build" -destination "platform=$platform,name=$device"
Like all professionals, we like to wrap our xcodebuild commands in Ruby, using a Fastlane script for our deployments:
lane :deploy do
match
gym
api_key = app_store_connect_api_key()
deliver(api_key: api_key)
endThis allows us to make far cuddlier GitHub Action workflows for automatic deployment, whenever we push and merge code into main:
on:
push:
branches: [ main ]
jobs:
deploy:
runs-on: macos-13
steps:
- uses: actions/checkout@v3
- uses: maxim-lobanov/setup-xcode@v1
with:
xcode-version: 15.1
- name: Create gc_keys # set secrets for `fastlane match`
run: echo $GOOGLE_CLOUD_KEYS >> gc_keys.json
- name: App Store Deploy
run: fastlane deployGitHub Actions also has an ecosystem of free and paid community-contributed workflows, which you can place seamlessly alongside your own commands.
The buck stops here with GitHub Actions. Setting up our deployment workflow and Fastlane scripts—including secrets, the Fastlane session, App Store credentials, and certificates—can be a complex task.
Fastlane match is a wonderful tool which takes away some of the pain of managing certificates and profiles on CI; but it’s a tricky beast to get set up which involves storing certificates on third-party cloud services.
While maximising control, the hands-off approach of GitHub Actions leads to 2-3 days of solid work for most experienced seniors just to get up and running. The big benefit, however, is a highly portable workflow which should run on any MacOS machine.
Setting up Appcircle
Appcircle is simultaneously much easier and more overwhelming.
Appcircle lives in a GUI, which contains modules to keep our ops team, mobile devs, and QA testers focused on their own domain.
Instead of spiriting me away from my repo to a hieroglyphical wall of xcodebuild commands, Appcircle begins with a blank canvas, and a helpful note asking me to connect a repo.
You can connect repos from GitHub, Bitbucket, Azure, GitLab, or even self-hosted repos via SSH.
Unfortunately, we can’t simply migrate from GitHub Actions by porting over our workflow files. Appcircle eschews the DevOps addiction to YAML and provides a straightforward GUI buffet of Configurations, Workflows, and Triggers.
Appcircle leans heavily on what-you-see-is-what-you-get: you configure your pipeline using drag-and-drop, assembling build steps directly in the GUI. DevOps purists can rest easy however: every one of these build steps is an open-source repo with a YAML configuration & shell script.
Appcircle also makes it simple to add third-party integrations as build steps such as Sonarqube, Detekt, Maestro Cloud, and Azure DevOps Bot.
The big win, however, comes in the form of managed certificates and provisioning profiles. Instead of setting up automation to retrieve these from third party storage, Appcircle can store, manage, and even create these internally.
My build worked first time without straying very far from the defaults.
We can set this to deploy automatically into App Store Connect or the Google Play Store with some brief setup in the Store Submission module of the dashboard.
Finally, Appcircle allows enterprise users to self-host, setting up a runner as a MacOS VM image to install in your own environment. This VM is fully managed, so infra teams won’t need to take on any of the painful versioning work.
Ease of setup evaluation
While GitHub Actions is substantially easier to find (at least, if you’re hosting on GitHub!), you are left to your own devices when it comes to setting up workflows. Your end result is a very portable set of automation scripts which should run anywhere, but dependencies on third-parties for certificate storage.
Appcircle offers sensible defaults, holds your hand through the pain-points of certificate management, and just works.
If I wanted to refine and estimate JIRA tickets for each approach (using the ubiquitous Fibonacci-numbered story points as a proxy for hours) they might look something like:
Set up CI/CD on GitHub Actions using Fastlane (21 points)
Set up CI/CD on Appcircle (2 points)
Getting Appcircle up and running is 10x simpler than GitHub Actions.
Infrastructure Management
All SAAS CI/CD services offer cloud-hosted build machines to run jobs, saving engineers from either the unpredictable sluggishness caused by local runs, and the janky sysadmin task of keeping your build machine alive.
Of course, some organisations—from highly regulated financial businesses to stealth startups protecting their IP—might want to self-host their CI/CD solution. Every org is different, and so supporting both approaches is table stakes for any enterprise-grade mobile DevOps solution.
GitHub Actions infra management
GitHub Actions offers all the basic options you might expect—Linux, Windows and Mac build machines. These images are also pretty seamless to set up through your workflow file—for example, to roll a custom MacOS machine (necessary for an iOS build), we simply set the runs-on property on our workflow job.
We can also take advantage of open-source workflows such as setup-xcode to select the right IDE version, so our deployment pipeline might look something like this:
jobs:
deploy:
runs-on: macos-13
steps:
- uses: maxim-lobanov/setup-xcode@v1
with:
xcode-version: 15.1Unfortunately, this open-source workflow is still only able to select Xcode versions pre-installed on MacOS cloud runner images.
We’re subject to the whims of GitHub project managers get to decide when the new build machines come out for OS releases: Sonoma machines aren’t even in beta yet. This can make GitHub Actions a bottleneck for your own iOS version target.
The versioning story gets even worse if you’re targeting iOS 17+. I’ve run into unexplained issues with SwiftData, Apple’s new declarative persistence framework, when running GitHub Actions workloads on the cloud.
These forces tend to drive most mobile GitHub Actions users to self-hosting—a process which usually involves petitioning your procurement team for additional Mac hardware.
Self-hosting is phenomenally easy to set up, and you can toggle your machine listeners via a simple shell command—sh actions-runner/run.sh. You can go further and configure your local machines to run the listener automatically in the background to handle your builds and deployments.
These are pretty simple to manage, via a system of tags which tell which workflow on which self-hosted machine it’s allowed to run. When rolling your own devices, you can have confidence they behave most of the time. This works great as long as you’re easily able to tinker with your build machine via screen-share.
If you need to keep your IP away from the cloud, GitHub offers GitHub Enterprise Server which can run on-prem using VMs hosted in your datacenter. This can be configured to support GitHub Actions, however there is no support for MacOS workloads—you will still need to set up a Mac machine as a self-hosted runner.
Appcircle infra management
Migrating to Appcircle is night and day.
In keeping with its do stuff that just works philosophy, Appcircle offers 2 nondescript drop-downs in the default CI configuration:
Machine pool—either Cloud Intel, Cloud ARM, or self-hosted
Xcode version (not required in Android builds)
We don’t need to manage OS versions, we don’t need to keep up with Xcode. There is a simple SLO for enterprise customers: the latest Xcode within 24 hours, including betas and release candidates. Today’s default M1 fleet runs MacOS Ventura 13.5.2, which is non-negotiable. This version is kept up-to-date alongside Xcode versions to ensure all OS versions are supported.
This management is really the killer app for Appcircle. Your infra team no longer has to muddle around with OS and Xcode versioning, because it’s all handled. This is even handled via the on-prem solution—Appcircle installs its own VM image.
Self-hosting has a similarly well-documented process to GitHub Actions—download the app locally, configure the runner, and set it to run your builds in the background.
Self-hosted runners have their own section in the build menu for simple management. Where Appcircle improves upon GitHub Actions, however, is how simple these machines are to use.
Instead of mucking around with sets of string-typed tags in your runner, and matching them up with your YAML workflow file, the Appcircle self-hosted machines in your organisation conveniently appear in the pipeline configuration dropdown.
Infrastructure management evaluation
While GitHub Actions gives you far more configurability, this falls over with the weight of enormous administrative and engineering complexity.
GitHub Actions’ hands-off approach to mobile CI creates a reliance on open-source contribution to fill in the gaps. Even worse, their sluggish approach to supporting new OS versions makes this process particularly painful on MacOS.
Appcircle takes on this OS and IDE versioning overhead itself.
With their plug-and-play build machines, and simple-to-operate self-hosted runners, this comparison isn’t even close.
Build Speed
This section pits GitHub Actions and Appcircle head-to-head on the speed of their flagship cloud-hosted Mac machines.
I support two, pretty standard, workflows in my project:
On Pull Request: I want to run my unit and UI tests before merging code.
On Push: I want to create a signed release build and submit to the App Store.
I’ll run both sets of workflows on each platform to ensure a fair test.
Benchmarking on my M1
To use as a baseline, I’m going to run my tests on my not-particularly-state-of-the-art 2020 M1 MacBook Pro, with 8GB RAM. It runs MacOS Sonoma (14.0) and Xcode 15.1.
This gives us a likely best-case scenario—a dedicated machine performing no other work.
The test suite ran in 2m 22s. That’s the time to beat.
Our deployment step runs lightning-fast, clocking in at just over a minute, 1m 2s.
GitHub Actions build speed
As mentioned before, GitHub Actions stays out of your way, serving as a pure runner for your build process. We need to sort everything out ourselves, but we get total control.
The test and deployment workflows are pretty simple to set up.
Our Fastlane script also does a good job of hiding away the complexities of xcodebuild to run my test plan and deployment.
platform :ios do
desc "Run tests on each PR"
lane :test do
scan(device: "iPhone 15 Pro")
end
desc "Deploy app to App Store Connect"
lane :deploy do
match(readonly: true)
api_key = app_store_connect_api_key(
key_id: "V4D62Q8UQB",
issuer_id: "69a6de92-2bb4-47e3-e053-5b8c7c11a4d1",
key_content: $APP_STORE_CONNECT_API_KEY_KEY,
is_key_content_base64: true
)
increment_build_number({
build_number: latest_testflight_build_number(api_key: api_key) + 1
})
gym(export_options: "./fastlane/ExportOptions.plist")
deliver(
api_key: api_key,
force: true,
skip_screenshots: true,
precheck_include_in_app_purchases: false
)
end
endThe key to using a cloud runner is runs-on: macos-13 to specify a GitHub Actions cloud runner with MacOS Sonoma (13.6.3). Setting xcode-version: 15.1 similarly ensures we’re on the right version of Xcode. Finally, we can simply run the Fastlane command in our workflow to call our automation script.
Overall the suite of unit and UI tests for a fresh build runs in a total of 8m 24s.
Setting up our deployment workflow and Fastlane scripts—including secrets, the Fastlane session, App Store credentials, and certificates via match—takes a long time, but results in a workflow which should run nicely on any Mac machine.
Unfortunately, on the macos-13 cloud runner, we sped past the default timeout for the workflow while it attempted to deploy our app—GitHub Actions got stuck on the code signing step.
Cloud-hosted MacOS runners on GitHub Actions are notorious for flakey performance and slow build times. Given they cost 10x as much per minute as Linux machines, you are frankly better off self-hosting.
A note on caching
Github Actions offers the ability to cache package dependencies and derived data. Derived data can speed up builds substantially, but often leads to flakey builds (delete derived data anyone?). Caching packages is a lot safer, but since this project only has 117kB of packages, I didn’t spend time implementing this for the sake of a couple of seconds.
Appcircle build speed
As discussed, Appcircle makes life a lot easier in terms of setup. You can give the configuration, workflow, and triggers setup UI to the most junior member of your team and likely end up with something that works well.
For iOS builds, Appcircle runs on MacOS Ventura 13.5.2. I selected Xcode 15.1 in my workflow configuration to ensure an equal comparison.
When first running our unit and UI test suite on a pull request, I was actually a little disappointed that this mobile specialist didn’t show a massive speed advantage above Github Actions - it took 5m 43s.
I was pleasantly surprised to discover that Appcircle had an automagical caching system kick in with my subsequent test builds, cutting runs down to 2m 25s.
Our push workflow, as discussed before, is phenomenally simple to set up—Appcircle takes on the vast majority of certificate management responsibilities.
There are a few stepping stones between the Build section which generates your signed builds, and the Testing Distribution and Store Submission sections of the Appcircle dashboard, but it’s simple to wire these up to automatically distribute test builds and deploy to the App Store each time your CI runs.
The end result?
It clocked in a little seconds slower than my local build, 1m 40s to produce my first, uncached, build.
Appcircle’s caching was a little less powerful this time around, but brought it down neatly to 1m 24s for my subsequent deployments.
Build speed evaluation
Speed is a critical part of the CI/CD workflow—when you have a bug affecting customers, every minute counts while you are deploying a hot-fix.
Appcircle is at least 32% more performant than Github Actions.
This drastically increases with caching and subsequent builds. Github Actions also suffers from inconsistency, making it difficult to recommend it on this metric—self-hosted runners are frankly a much better fit for mobile workloads.
Permissions Management
Permissions management and access control are mission-critical concerns with any big project. You don’t want to simply give a disgruntled intern the keys to the kingdom.
Between setting up workflows, managing testers, to entriely segregated set of permissions managed on App Store Connect and the Google Play Console, mobile apps consistently constitute a pain point here for enterprise ops teams.
GitHub permissions management
GitHub Actions is itself a feature of GitHub, so we can’t discuss permissions management in Actions without understanding how it works in GitHub as a whole.
GitHub’s top-level collaborative entity is an Organisation. These contain a fractal pattern of permissions where nested hierarchies of Teams have their own sets of permissions, as can individuals.
This complex org structure is simplified on the permission level with a few granular, broad-brush options for Teams and individual contributors.
These permissions levels range from Admins—von Neumann machines that clone repos and add more collaborators—to Readers, who look but don’t touch.
GitHub also allows Enterprise users to create up to 3 custom repository roles with 40 individually configurable permissions.
How does this relate to GitHub Actions?
Since you are setting up your workflow entirely in code, sitting in the nondescript .github folder at the top-level, repo access permissions are GitHub Actions permissions. Those YAML workflows are editable by anybody with permissions to push code to your repository.
App tester management isn’t a consideration here at all, since GitHub Actions doesn’t natively support this. You are required to head over to App Store Connect and the Google Play Console in order to manage permissions pertaining to app management.
Appcircle permissions management
Appcircle also uses Organisations as the top-level entity. Users own an individual organisation by default, and can create and join organisations to collaborate in their company workspace—all under the same email address.
Enterprise users may also create sub-organisations, which operate exactly the same as the top-level org except with their own managed sandbox of permissions, apps, pipelines, and testers.
For enterprise users, this two-tier approach far more basic than GitHub’s nested hierarchies—however the low-level permissions management is comprehensive.
When adding a new user to the platform, you can set their organisation, as well as a comprehensive menu of fine-grained permissions.
These permissions range across the entire end-to-end solution and include builds, environment variables, code signing, provisioning profiles, app testing groups, app store submissions, publishing apps, organisation management, enterprise store management, billing, and any third-party connections.
Appcircle is designed to keep people in their lanes, so to speak:
Mobile engineers set up the certs and profiles. Ops can configure the pipelines. QA can set up and auto-deploy test builds in the GUI. Even marketing can get in on the action to handle App Store approval. No user needs any more or less access than they need.
This is phenomenally granular, however the downside against GitHub lies in pure admin load—since you can’t simply assign role-based access to teams, each user must be set up and managed individually.
Permissions management evaluation
GitHub Actions and Appcircle take differing approaches to permissions management and access control.
The best option here really depends on your requirements.
GitHub’s simple role-based access control for teams is a good fit when you have teams of engineers who require different levels of access to specific repos—for example, you might us this when applying an internal open source model where the Android team can see the iOS repo and review PRs, but can’t push code. The lack of granularity doesn’t really hurt because GitHub is only used by engineers—you might be using different solutions for other functions. The enterprise limit of 3 custom roles is restrictive if you really need something customised.
Appcircle is untouchable if you are utilising all its features with specialists across your organisation—engineers, testers, ops and marketing get fine-grained access to exactly what they need, and no more. Sub-organisations make management simple even for extensive app portfolios. The main downside is a higher admin load when setting up each user.
Unfair Advantages
There are several metrics on which it’s unfair to compare GitHub Actions with Appcircle, since Appcircle encompasses far more than just building your code.
As an end-to-end solution, Appcircle is designed for mobile devs first, yes, but also ops, testing, and marketing. There are features which GitHub Actions simply isn’t designed to do.
#1: Tester management and deployment
This is really where Appcircle comes into its own as an end-to-end solution, since your QA team and even external testers will see instant quality-of-life improvements.
It’s trivial to set up your pipeline to send builds to tester groups—for example, for internal QA or external beta testing.
Apple test devices can be fetched from the Apple Developer portal, and managed locally in Appcircle via the Testing Distribution module of the dashboard. This uses App Store Connect APIs under the hood, but you can manage everything, including adding new devices, from inside Appcircle.
It’s easy to assign test groups with specific users. For each app in your organisation, you can also automatically push out new builds to these groups.
You can set up public test links for open betas, or keep alpha builds on lockdown via a username and password. Appcircle also supports SSO and LDAP for authentication, so most enterprises should be able to use their preferred approach to locking down test builds.
It’s trivial to configure your CI workflows to automatically distribute test builds. Appcircle does this blazingly fast, with a total of 1m 50s between starting my deploy workflow and sending me the link to the test build via email.
GitHub Actions doesn’t even attempt to compete in the test space: you need to set up Fastlane scripts or configure automatic beta distributions on App Store Connect and the Play Store Console. Before you can download these, there is often processing which you need to wait for—this can vary enormously, but is often around 10 minutes.
Therefore Appcircle is about 6x times faster at distributing to testers.
#2: Certificate and profile management
Code signing is the part of mobile CI most likely to send a DevOps engineer into anaphylactic shock. It’s a gauntlet that every mobile engineer runs through in their transition from junior to senior.
Appcircle puts a great deal of effort into painlessly guiding you through this process. There is an entire Signing Identities module on the dashboard dedicated to managing certificates, profiles, and keystores. These help guide you through the process of creating and securely storing your App Store Connect API key; creating a certificate signing request; making your dev and prod certificates; and signing the necessary provisioning profiles.
From zero to profiles, I had to spend approximately 3 minutes in total on App Store Connect.
After storing your profiles in Appcircle, the provisioning profiles conveniently appear in drop-downs in the signing configuration step of your pipeline. You can even ask Appcircle to automatically produce profiles with your certificates.
This process is semi-possible to automate via GitHub Actions. You will usually end up using fastlane match to create your certificates and securely store them; and configure a secret keys file in the build step to fetch them each time you build.
These additional steps constitute the biggest difference in ease of setup between the services—most importantly, setting up third party storage to securely host your certificates. Properly setting up certificate management for GitHub Actions via Fastlane is about a day of work to Appcircle’s (approximately) 15 minutes.
#3: Enterprise App Store
This is a fairly unique addition to Appcircle that satisfies a very specific enterprise use case—distributing an app internally to your own employees.
Internally distributing an app in a large company apps requires navigating a labyrinthine web of sponsors, steering groups, and protective ops teams—a pain I would not wish on my worst enemies.
Appcircle has their own solution for this nightmare, allowing you to host your own private enterprise App Store.
GitHub Actions doesn’t even touch this functionality, but, alongside a bought-in ops team, this feature will be a lifesaver to many.
#4: Automating with APIs
This is actually a point in favour of GitHub Actions. Since it’s a far more mature solution, targeted squarely at the ops director, functionality from access control to billing and everything in between is accessible through the GitHub REST API.
Virtually everything is automatable in code via GitHub’s API.
This, alongside the ability to place your entire CI in a set of YAML files, makes GitHub an attractive option for sysadmin types who rarely leave the terminal.
Appcircle does have a CLI, however as you might expect it’s far less powerful than its counterpart.
Unfair advantages evaluation
Overall, GitHub Actions is clearly a more mature solution, however it’s designed to work for a general software engineering workload. It’s a blank slate—a configurable, agnostic, workload execution environment.
For mobile use cases, GitHub Actions probably does about 40% of the things Appcircle can do—a result of it being a general tool rather than a specialist mobile solution. Therefore, many of these unfair advantages come about entirely due to Appcircle’s focus on the enterprise mobile use case.
Conclusion
GitHub Actions is a mature product in a mature business. In 5 years’ time, it’s not likely to look much different. Basically, it works fine. Today’s mobile engineers are usually asked to accept the status quo, stop being difficult, and find a way to get the CI to work.
Appcircle just works.
There’s one more benefit of Appcircle I’ve not yet mentioned: The future. Appcircle is only a few years old, and growing fast.
In the next 6 months, they intend to bring their Publish module out of beta, capturing more and more of the App Store Connect API surface—enabling full cycle release management without leaving the dashboard. It’s also delving into AI tooling to fix common pain-points in the testing and release workflow.
I’m excited to integrate Appcircle into my projects going forwards, and recommend the same for you.