Schema migration is a daunting process but you must get it right. Otherwise you might end up breaking your app and corrupting user data in the process. This articles aims to make custom Core Data migrations more approachable.
When dealing with something as critical you want to keep things simple. With Core Data it means either avoiding migrations altogether, or using lightweight migrations. However, sometimes custom migrations are unavoidable.
This part describes how you might encounter custom migrations for the first time, and what are progressive migrations.
Imagine you are working on an app. You started with mom_v1 (NSManagedObjectModel *version 1). Later you added mom_v2 with minimal schema changes. At this point your Core Data setup might look like this:
When you add a store that uses mom_v1 to the NSPersistentStoreCoordinator, it automatically creates an inferred mapping model between mom_v1 and mom_v2 and performs a lightweight migration.
Imagine you now need to add mom_v3 with changes that can’t be carried out by lightweight migration - you need a custom mapping model from mom_v2. You jump through all the hoops required to create one. But then comes the problem. How does NSPersistentStoreCoordinator works with custom and inferred mappings?
Here are some pieces of documentation that we get:
If the version hash information for the added store is determined to be incompatible with the model for the coordinator, Core Data will attempt to locate the source and mapping models in the application bundles, and perform a migration.
When combined with NSMigratePersistentStoresAutomaticallyOption, coordinator will attempt to infer a mapping model if none can be found.
After some experimentation you find out that:
When you add a store with mom_v2 to coordinator it finds your custom mapping mom_v2 ~> mom_v3 and uses it for a migration (as expected)
When you add a store with mom_v1 to coordinator it looks for a custom mapping mom_v1 ~> mom_v3 which it can’t find. It then infers a mapping model and performs a lightweight migration.
The second case worked not how you might have expected, or at least not how you wanted it to. What we want is to migrate from mom_v1 to mom_v2 using an inferred mapping, and then migrate from mom_v2 to mom_v3 using a custom mapping - perform a progressive migration.
Unfortunately, Core Data doesn’t support progressive migrations like this out of the box - via some boolean option. Instead it optimizes for performance by skipping intermediate steps. In fact, there is no notion of linear version history built into Core Data at all. While it makes sense for lightweight migrations, it doesn’t really work for custom ones.
It might be tempting to create yet another custom mapping mom_v1 ~> mom_v3 to workaround this problem, but that would be an embarrassment. It obviously doesn’t scale well as we add new model versions. Fortunately, there are at least two different ways to implement progressive migrations using Core Data APIs.
Progressive Migrations (Automatic)
There is a very simple way to trick Core Data into performing progressive migrations using only NSPersistentStoreCoordinator. Each coordinator is initialized with a destination mom. When you add a store to the coordinator it performs a migration to the mom that it was initialized with. A destination mom doesn’t have to be your final mom version. We can use these facts to implement progressive migrations:
Here’s a slightly modified non-recursive implementation of migrate function:
This function takes a linear version history as a parameter. To create it you should read all moms from the bundle and order them. Compiled moms have .mom extension and are in general stored in a subdirectory with .momd extension. To create a list of moms you can either provide an ordered list of hard-coded file names, or implement some simple heuristic that reads and sorts them automatically. I prefer the former, because mom names come in handy when writing tests.
And here’s an implementation of indexOfCompatibleMom(at:moms:) function:
This simple solution does just what we wanted, but it has some downsides:
Core Data looks for mapping models in bundles returned by NSBundle’s allBundles and allFrameworks methods. It can’t search for mappings in other locations, which can hurt testing.
Eventually, you might want to customize a migration process.
Progressive Migrations (Manual)
The algorithm is similar to the one introduced in the previous part, but this time we use NSMigrationManager class to perform migrations. Here’s an outline of what we need to do:
You can find a full non-recursive implementation here. The provided functions match the ones defined in the pseudocode. Let’s dive into the code to see some of the details.
Here’s the first function that loops through the object models. It uses an indexOfCompatibleMom(at:moms:) function that we added earlier.
Next is a function that performs an actual migration. As you can see it relies heavily on Swift error handling model.
The last step where we replace the source store is very important. The replacePersistentStore(at:...) method (which was added in iOS 9.0) honors file locks, journal files, journaling modes, and other intricacies related to SQLite. It guarantees that you won’t corrupt user data by failing to replace stores in a single transaction.
The last piece is a simple function that finds a mapping model:
Testing migrations is extremely important. It makes sense to test that migrations are performed progressively first. And then test each migration step (mom_v2 ~> mom_v3, etc) in isolation.
It is very important to test migrations on different data sets and different versions of your store. There are at least two ways to populate stores with data:
Run a specific version of an app. This way you generate data just like your app actually does, including some potential errors that you might be not aware of.
Populate stores programatically. This would allow you to simulate some very specific cases that are either hard or impossible to reproduce on the device.
Here’s how your test methods might actually look like (this is a very rough draft):
You can skip several moms during migration to improve performance. While it makes sense for lightweight migrations, it doesn’t really work for custom migrations. In general it is ill-advised, because it complicates the migration process, and it doesn’t really come into play unless users skip multiple versions of your app.
If the migration fails you might want to delete (or copy to a safe location) the existing store and create a new one using the latest mom.
Multi-pass migrations can be used to migrate large data sets without using too much memory.
Mapping model becomes invalid after changes to either its source or destination moms. You have to recreate a mapping model after you make such changes.
If you set the com.apple.CoreData.MigrationDebug preference to 1, Core Data will log information about exceptional cases as it migrates data.
The com.apple.CoreData.SQLDebug preference lets you see the actual SQL sent to SQLite.