Total Control of Your Camera Using Pixel Buffer APIs

Come for the AVCaptureSession, stay for the Core ML & Metal

The iPhone camera is an iceberg 🧊

Virtually everybody’s used the basic APIs at some point to capture media or show the camera preview on-screen.

But there are decades of APIs beneath the surface.

The Camera and AVFoundation APIs have evolved into an unbelievably sophisticated toolkit that dual-wields powerful sensor hardware alongside ancient CoreMedia SDKs to give you total control over every pixel.

In modern iOS development, this means we can leverage Metal or Core ML to manipulate reality in real-time.

When I first saw these techniques in practice by our resident Camera and shader expert, I thought he was a god. Since being blessed with this wisdom, I’ve levelled up a little on the divine corporate hierarchy. I reckon I’d make a kickass prophet.

So. That’s what we’re learning today:

• We’ll begin by reviewing the basics of AVFoundation.

• Next we’ll make the most basic possible Camera implementation using AVCaptureSession and AVCaptureVideoPreviewLayer.

• Next we’ll invoke AVCaptureVideoDataOutputSampleBufferDelegate to access the individual frames from our camera.

• Finally, I’ll show you how to transform this data via Metal or Vision to create real-time camera effects that look a little like this:

By the end, you’ll be ready to implement these tools in your own apps, and win a seat among the pantheon of camera saints.

Contents

• AVFoundation Primer

• Building a Simple Camera

• Accessing the Pixel Buffers

• Real-time Image Processing

  • …with Metal

  • …with Vision

• Last Orders

Subscribe to Jacob’s Tech Tavern for free to read ludicrously in-depth articles on iOS, Swift, tech, & indie projects in your inbox every week.

Paid subscribers get much more:

• Read this article, plus all my Elite Hacks, my exclusive advanced content 🌟

• Get my free articles 3 weeks before anyone else 🚀

• Access my full Swift Concurrency course 🧵