The camera is the prime differentiator that elevates mobile experiences above the web. Without it, the biggest social apps today such as TikTok, Snapchat, and Instagram simply wouldn’t work.
In the early days of smartphones, cameras were crappy, hardware was slow, and there were no first-party image processing tools. Clever devs used filters to make photos look better — writing custom graphics shaders with OpenGL to eke out what they could from a 2010 phone CPU.
Life is different today: cameras are professional-grade, processors are screamingly fast, and high-level APIs make it easier than ever to process images in your app.
With modern tech comes modern challenges — with the camera app market so saturated, photos on your app need to look better than the rest of the competition.
On iOS, knowing Core Image is the first stage of this differentiation.
Today I’ll explain how Core Image works, demonstrate how you can use it in your apps today, and discuss the many system filters.
If you’re impatient, and not averse to spoilers, you can download my CoreImageToy app on the App Store and check out the open-source project on GitHub.
Core Image
Core Image was introduced way back in 2005 with OS X Tiger, but wasn’t ported to iOS until iOS 5 in 2011.
Core Image was designed to offer a simple API for high-performance image processing — applying multi-core parallelism to transform each pixel without you, the developer, needing to worry about it.
Core Image is based around Filters, which individually define some kind of processing for an image — for example, adding a blur effect, adjusting the colour, or distorting its geometry.
Filters can be chained together, meaning you can combine filters. For example, you can chain a sepia colour filter, to a Gaussian blur, to a glass-lens distortion.
Core Image is also designed to be pluggable, meaning that you can create your own custom CIFilters which integrate seamlessly with the filters bundled in the framework. Whoever said object-oriented programming is dead, eh?
Core Image Performance
Core Video is actually implemented using Core Image. To ensure that Apple could handle real-time video processing (even in 2005!), Apple applied several techniques to ensure sure it could render each frame in under 1/60th of a second (or whatever the frame-rate du jour is).
Filters and Parallelism
What is an image?
It’s essentially a big ol’ grid of pixels.
A basic colour filter is conceptually very simple:
Take the RGBA colour for a pixel, perform some mathematics, and then return a modified colour.
Now do the same for every single pixel.
There are more complex categories of filter, such as a warp filter (which moves the pixels around), or a sampling filter (which transform a pixel based on the colours of neighbouring pixels). But the concept of “do stuff to each pixel” is the same.
Consider creating your own image processing library.
You might start your proof-of-concept with a naïve implementation, iterating through all the pixels of an image and processing it, applying the resulting colour to a new image (or grid of pixels) at the end.
for x in image.width {
for y in image.height {
let originalColor = image.pixel(x, y)
let newColor = process(pixel: originalColor)
newImage[x][y] = newColor
}
}Notice the problem?
Running this algorithm in series on a CPU core is like sucking a swimming pool through a straw. It feels pretty fast when you’re the guy doing the sucking, but the job is way too big to be efficient.
It is an operation begging to run in parallel.
Hardware Acceleration
Even if we parallelised across the 6 CPU cores in a modern iPhone, it’ll still be pretty slow (and guarantee we clog up our main thread).
Core Image automatically targets the fastest hardware available — back in 2005, this depended on how your Mac was configured, but today this is basically always going to be the GPU.
If you want to impress your co-workers, you can refer to this as hardware acceleration (in general terms, this is whenever a task is handed off from the CPU to specialised hardware).
The GPU of an iPhone 15 Pro has 8 individual cores with 128 total execution units, which can run the simple mathematical operations like they’re nothing.
When we run our naïve pixel-processing operation on the GPU, it’ll run simultaneously for each pixel (well, 128 of them run simultaneously), and might look a little like this:
float4 process(float4 pixel) {
return process(pixel);
}This is more akin to you and 127 of the lads emptying the pool out with buckets. Core Image manages all this parallelism internally, and its high-level API means developers don’t even need to think about it.
Lazy Evaluation
You might expect multiple chained filters to perform work in an O(n) fashion, losing performance with each subsequent pass as more processing steps are run, and the memory I/O of intermediate products increases.
Core Image actually implements a clever lazy evaluation, which allows performance of multiple chained filters to (often) work just as fast as individual filters.
The way this works is actually pretty interesting.
The CIFilters themselves are treated as steps in a recipe, with CIImages set up as input and output products for each filter. All this wiring up doesn’t actually perform any work: the processing is lazy, only applied when the final image is requested.
The filter steps and inputs are linked together as a graph data structure. When rendering, Core Image uses a just-in-time compiler to convert this graph into a pixel shader which implements the whole filter chain, which runs on the GPU.
What on Earth is a shader?
I’m really glad you asked.
I’m honestly going to go into a lot of detail on shaders very soon, but if you can’t wait, a shader is essentially a little function that runs on the GPU. They are called shaders because back in the olden days, before colour computer screens were commonplace, they controlled the shading of shapes rendered on-screen.
Read Metal in SwiftUI: How to write Shaders for a detailed, beginner-friendly introduction.
Using Core Image
You can utilise Core Image filters in your apps quite easily. Let’s set up a sample app with, say, a photo of your cat.
(dog people can exit the article now)
Once we add an image to our asset library, it’s not a lot of code to display it in the default SwiftUI app ContentView.
struct ContentView: View {
@State private var viewModel = ContentViewModel()
var body: some View {
Image(uiImage: viewModel.uiImage ?? UIImage())
.resizable()
.aspectRatio(contentMode: .fit)
Button("Apply Filter") {
viewModel.applyFilter()
}
.padding()
}
}
Our creatively-named ContentViewModel is where the magic happens. Let’s go from the top:
CIFilterBuiltins
// ContentViewModel.swift
import CoreImage
import CoreImage.CIFilterBuiltins
import SwiftUIWe’re going to need CoreImage, of course, but CIFilterBuiltins goes a long way towards avoiding Objective-C-era cruft.
Basically, instead of instantiating CIFilters with string-ly typed names:
filter = CIFilter(name: "CIThermal")These builtins offer a type-safe alternative.
filter = CIFilter.thermal()The View Model
Next, we have a few properties on the view model.
// ContentViewModel.swift
@Observable
final class ContentViewModel {
private let context = CIContext()
private let originalImage: UIImage?
var uiImage: UIImage?
var filter: CIFilter?
init() {
originalImage = UIImage(named: "cody")
uiImage = originalImage
filter = .thermal()
}
// ...
}The view is displaying the uiImage property, which we will modify with our chosen CIFilter.
I left filter as an internal property so you can play with it later.