This is my most elaborate article yet, covering my full journey building my new indie project, Summon Self, from conception to initial release.
This app turns you into a trading card, with personalised stats, graphics, typing, and rarity. You can trade these cards IRL to create a full collection of your friends.
I applied client-side AI models using MobileCLIP, CoreML, and on-device image classification to generate personalised stats from your own photos.
I combined these with Metal Shaders and the Vision Framework to turn you into a trading card, including graphical effects across 4 different card rarities: Common to Secret Rare.
I pulled everything together using SwiftUI, Swift Concurrency, and SwiftData.
Download Summon Self on the App Store today!
Share your favourite creations and your rarest cards, either in the comments or on the Twitter/LinkedIn/BlueSky threads!
As a 90s kid, I was the perfect demographic to get sucked into the early 2000’s Yu-Gi-Oh craze. I had a deep fascination with Pegasus*, the foppish creator of the card game, who possessed a magic eye that could trap people’s souls in trading cards.
Since watching the anime at age seven, I’ve been fixated on the feeling of power that trapping someone’s soul in a trading card must bring. Relentlessly. Like Kaiba, I mastered cutting-edge technologies in order to achieve my objectives.
Today, client-side artificial intelligence brings me all the tools I need to make this dream a reality.
I have a simple, beautiful, vision:
My app turns you into a trading card, with personalised stats, graphics, typing, and rarity. You can trade these cards IRL to create a full collection of your friends: a social trading-card binder.
Today we’re going to follow my journey bringing this vision alive: from concept, to POC, to MVP, to the App Store.
*Unfortunately, with Pegasus, I backed the wrong horse entirely. My wife, also a big millennial Yu-Gi-Oh fan, actually fancied Marik.
The Concept
I keep my indie projects simple.
Three core features.
First, we take a photo of you to convert into a trading card stored on-device. The card has a personalised title and stat ratings based on your own attributes, detected on the image.
Second, we need some neat graphics so we can create a precious-looking rare trading card out of your photo and stats.
Finally, this card should be tradable with friends. This has to happen in person—you boof your iPhones together and get each other’s souls cards.
The build itself should be pretty straightforward: I’m putting together several moving parts that I’m already familiar with. The innovation is in the novel combination.
The important, boring, basics
Photos, persistence, and presentation are all bread-and-butter SwiftUI and SwiftData. Nothing special here, since I don’t plan to need more than 2 or 3 screens.
Stats
I want to upload the photo of my above Pegasus cosplay and be given a fitting title, such as Jacob the Creator, or Jacob the Illusionist.
Perhaps the stat-line will read:
Whimsy: 3000
Dapperness: 4600
Magic: 2100
Charisma: 4000
There is a way to deterministically generate stats like these and titles based on photos—this is a category of machine learning known as zero-shot image categorisation. Apple has created client-hosted AI models which we can customise with our own text categories, allowing us to scientifically create accurate stats:
Because it’s an ML model giving the stats, they must be true.
Looksmaxxing
I have a long-dormant idea in my head to make the cards look fantastic, rare, and shareable. I want to adopt the graphics tools used in this Pokémon CSS demo website and bring it into an app. I might host some custom CSS in an embedded browser, or perhaps I’ll roll my own SwiftUI Metal shaders.
There’s also something I plan with the iOS Vision framework. This is the second piece of client-side AI tech that will make these cards shine.
Catching ‘em all
Whoops, wrong franchise. This is the cherry on the cake that will make this the biggest viral hit since Pokémon Go. Ever since WWDC 2023 introduced SharePlay via Airdrop I’ve wanted to build a local P2P trading card game.
The concept is straightforward: boof iPhones together to set up a SharePlay session and trade Transferable card objects.
Now we’ve established the plan. It’s time to execute.
Implementing CLIP & client-side AI
I hope you’ll forgive me for skipping past the SwiftUI basics that are of interest to nobody. I vibe-coded the basic structure so we can get to the interesting stuff.
CLIP, or Contrastive Language–Image Pretraining, is an image classification model trained on both images and text. It involves 2 specialised models: an image model that converts images to vectors, and a text model that converts language into vectors. By multiplying these vectors, we work out which text categories are ‘conceptually closest’ to the image in vector space.
I’m going to do a separate deep dive on how to make your own CLIP models next week. The most important thing to know is I’m using MobileCLIP BLT, a CLIP model trained, bundled to CoreML, and open-sourced by Apple.
Embeddings
I ran a Python script running on the MobileCLIP text model to generate embeddings for all possible stats and titles. These are 500-item arrays that contain a representation of the image, or piece of text, as a point on 500-dimensional vector space.
Here’s the vector embedding for the Attack stat.
And here’s some of the vector embeddings I’ll use for card titles.
Now that I have the embeddings, I don’t need to bundle the text model, since the only variable is the image you upload. I can just bundle the image model and generate the required image embeddings.
CoreML and Image Processing
In my code, we can use the Vision framework to handle the CoreML image model on our iDevices. First we load in the bundled models and embeddings:
To make things less verbose, I am redacting the code a little and force-unwrapping everything. I have tons of error handling in the real version, Knight’s honour 🤞
Next, we can process images with Vision and this CLIP model:
Finally we make predictions using the CLIP model and text embeddings:
We multiply each title embedding vector with the image embedding to return a confidence number. Whichever number is highest is the closest “match” to the image.
This name is the user’s entered name, and the title embedding JSON literally uses “X the great”, so we simply replace X with the name. Jacob becomes Jacob the Total Chad*
*I’ve unfortunately not yet found a CLIP model that reliably rates me a chad
We do the same for stats, and pick the highest few, to generate a fully customised stat-line. This uses the actual confidence numbers from our vector dot-products here to give a number for each stat that users can compare with each other.
Remember the law of large numbers:
Large numbers are cool.
Therefore I’ll multiply all these normalised numbers by about 10,000 to transform our ratings from Attack: 0.7400 to Attack: 7,400.
As well as their top custom stats, everyone gets the trading card staples of attack, defence, and magic.
The Proof-of-Concept
The best part of all this CLIP image classification? This is all deterministic!
Therefore the same photo will return the same results every time. Even better, since the image and text models are trained together in conceptual space, similar images will give similar results!
We now have enough to trial a proof-of-concept.
The numbers could do with some tweaking, but the names and stats fit.
This works.
It’s pretty fun already, and me and my friend were playing with it all evening. There’s a spark here. Now let’s make it sparkle.
Trading Card Graphics
We’ve got our cards, our titles, and some stats.
It’s time to play around with the view to make it into a tradable, shareable, shiny, rare trading card.
The inspiration for this came heavily from this Pokémon card CSS demo, which I’ve jealously played with for years.
I want to present you as a beautiful holo-foil rare card.
For a pre-proof of concept, I cheekily inspected the HTML on the page to paste in my own photo. It looks great!
But the holographic radiant effect on tilting the image doesn’t quite fit, since it’s customised for the original photo. I need to be cleverer with masking. But the Vision framework has a solution.
My original plan was to fork this repo, place your card in a web-view, and present your image via a filepath. However, the copy-left license and lack of accelerometer data made this a little awkward, so I looked for an iOS library that might serve as a starting point.