← iOS

Updating the Managed Object Model

Read This First

There are overarching things about managed object model (MOM) work that everyone must understand before doing the work.

Consolidating Changes

First and foremost, it is vital to understand that we never publish more than one new MOM per release. There are multiple reasons for this approach, but the two most important are the following:

  1. Minimize the impact felt by end users on the first launch after updating from the App Store.
  2. Reduce the total number of accumulated MOM versions (which plays directly into the above).

There are consequences from this decision. The biggest is that we must coordinate on what version will be used for the release under development. A version of the MOM that has been published to the App Store must never be modified again. Hence, we need to know whether a change needs to go into an entirely new MOM version or into an unpublished MOM version.

Second, it is quite painful to make multiple changes to the in-progress MOM version during a development cycle. Hence, it is very important for everyone to understand what it means to make changes to an existing MOM version.

In years past, this pain was typically avoided by doing all the MOM changes in one big PR at the start of the development cycle. That required knowing up front what all the required changes would be, and that does not always work out. However, the more changes that can go into one PR early in the cycle, the better. It makes for one big PR that is tedious to create instead of multiple big PRs that are tedious to create.

Iterative Migration

Lastly, we use an iterative migration approach that requires a heavyweight migration from published version N to newly published version N+1. The technique we used is based on what Marcus Zarra presented in his Core Data book and that he has also promoted on Stack Overflow. What it does is guarantee that every user always gets the same persistent store migration path regardless of the previous version of the app that was installed. If a user has V004 of the mapping model and needs to get to V007, the migration will go from V004 to V005 to V006 to V007. If a user has V005 and needs to get to V007, the migration will go from V005 to V006 to V007.

Adding a New Version

  1. In Xcode, select Banno.xcdatamodeld in the project hierarchy. This is found under the Banno group in the Banno project.
  2. Select Editor -> Add Model Version… in the menu.
  3. Click Finish.
  4. Select the newly added .xcdatamodel item under Banno.xcdatamodeld in the project hierarchy.
  5. In the inspector view, make sure that the File inspector is selected.
  6. Under Model Version in the File inspector, make the newly added version be the current version.
  7. Add properties to or remove properties from the relevant Managed* classes based on additions and removals of attributes and/or relationships in the new MOM version.
  8. Add or remove class methods corresponding to the property changes to provide the string identifier for a given attribute (someAttributeName) or relationship (someRelationshipName)
  9. Add the necessary subclasses of NSManagedObject if new entities have been added to the MOM. The base class is hard to predict, but the there are three likely cases:
    1. ManagedBannoObject: Used for a Banno API data model that has a unique identifier.
    2. NSManagedObject: Used for an API data model that does not have a unique identifier.
    3. ManagedSyncableObject: Used for a Banno API data model that has a unique identifier and that supports offline editing. As offline editing support is in decline, this is unlikely to be the correct choice.
  10. In the Xcode Project navigator, expand the group under Banno/Banno/MappingModels/BannoA_B.xcmappingmodel.
  11. Add a new Core Data mapping model to the end of the list that will map from the previous version to the newly added version.
  12. Apply all relevant customizations to the newly added mapping model.
  13. Add a new test case class to the MigrationTests target. See below for more information on this.

Modifying an Existing Version

IMPORTANT: Modifying an existing MOM version must be coordinated with release planning. The published/unpublished state of the MOM depends on what has happened on the develop branch since the most recent release was tagged. The best way to determine what to do is to ask.

If a change must be made to an unpublished MOM version B, then the mapping model from A to B must be recreated. There seems to be no way to get Xcode to “refresh” the existing mapping model to account for in-place changes. This is terribly annoying and time-consuming.

  1. Make a note of all customizations present in BannoA_B.xcmappingmodel. Every single one will have to be re-applied after the mapping model is recreated. Order of migration is typically not important, but it can be. It is important to confirm whether the order of the migration steps needs to be restored as Xcode seems to produce a random ordering. If it will need to be restored, make a note of what the existing order is.
  2. Delete the existing mapping model at Banno/Banno/MappingModels/BannoA_B.xcmappingmodel.
  3. Recreate the mapping model from A to B using the same name and at the same place in the project folder structure.
  4. Re-apply the customizations, including ordering if necessary, to the recreated mapping model.
  5. Apply any additional customizations needed that pertain to the new changes that led to the recreation.
  6. Apply any relevant additions to code that is used by MigrateAtoBTests.swift. Typically, this will entail making changes to graphB.json, EntitySourceVB.swift, and MigrationExpectationsVB.swift under Banno/MigrationTests.
  7. Change the target to MigrationTests and run the tests in MigrateAtoBTests.swift.
  8. Stage and commit the updated BannoA_B.xcmappingmodel. The changes to Banno/Banno.xcodeproj/project.pbxproj can be discarded unless other files were added or removed for some reason. (That should not be the case unless something really unusual has happened.)
  9. Stage and commit any updates under Banno/MigrationTests.

New Interfaces for New Entities

When adding a new entity to the MOM, there are an assortment of interfaces that typically have to be added. Persistence is based on data models defined by the Banno API and rendered for client consumption as JSON. For the purposes of this example, call the new entity Example.

Required

The only thing that is truly required is an NSManagedObject subclass, written in Swift, for the Example entity. This would typically be called ManagedExample, and it should be declared as a final class. It is okay to exclude attributes or relationships from the public interface if that is appropriate.

Optional

The actual work for persisting the received JSON to Core Data depends on the project requirements. The most common approach is incorporate the following:

  • An extension to the Decodable type that adds conformance to CoreDataSerializable. This may or may not make sense for the case in question. It does offer a good way to encapsulate the persistence work “close to” the Decodable type, though.
  • An extension to the HTTPClient.HTTPRequest type that adds conformance to CoreDataSerializableHTTPRequest. How this is implemented varies. In some cases, the body of the extension can be empty because the default implementation covers exactly what is needed. An example of this can be found in AccountGetRequest.swift. The persistence work is done in Account+CoreDataSerializable.swift. What is more common, though, is some sort of custom logic on the response payload that then routes into the code that performs the persistence work.

The interaction between the components that, respectively, conform to CoreDataSerializableHTTPRequest and CoreDataSerializable varies. Ideally, there will be some separation of concerns so that the persistence work can be isolated from the work to get the response payload into shape for persistence.

IMPORTANT: Try to minimize saving the managed object context as much as possible. For example, do not use performWithAutoSave(schedule:_:) if it is possible that nothing will be saved. As just one example among many, if the response payload may contain an empty array, use a guard to confirm that the array is not empty before doing the persistence work and saving the context.

Summary

All of that being said, none of the above is required except the class identified in the MOM for the Example entity. Performing the API request and performing the Core Data inserts, updates, and deletions can all be done in whatever way is suitable. What is described above represents how things are done for data models such as accounts, transactions, and transfers. Ultimately, it tends to end up as a lot of very small types that are wired together by higher level code.

Testing

There is a special test suite in the project just for MOM migrations. This is called MigrationTests, and it is only executed when made the active target. This test suite is fairly slow because it performs on-disk persistent store setup, migration, and validation. Running the whole suite can take a while, and this underscores why we never publish more than one new MOM version per App Store release.

The testing is based on a series of steps that are conceptually straightforward:

  1. Open a Core Data stack using MOM version N.
  2. Insert entities into the context.
  3. Complete the object graph by applying relationships between those entities.
  4. Write the object graph to disk using Encrypted Core Data with full encryption enabled.
  5. Perform the migration from version N to N+1.
  6. Open the migrated persistent store using MOM version N+1.
  7. Confirm that the migration had the expected effect.

In practical terms, this is vastly easier said than done. None of the NSManagedObject classes can be used in any way. This is because the current version of those classes must exactly match the current version of the MOM. However, the migration tests must represent every possible step in the iterative migration going back years to V001. Therefore, everything has to be based on string identifiers.

The Swift test code is based on using a sort of object graph described in a JSON file. The JSON captures all the attributes and relationships for each entity that will be inserted into the object graph for testing. The JSON can have zero or more “instances” of a given entity, and both to-one and to-many relationships are supported. Thus, it is very easy to get variety in the object graph that will be migrated.

For each entity in the MOM, there is a corresponding struct that conforms to Decodable that can be instantiated by loading the JSON file. How a given struct is defined depends entirely on how the corresponding entity is represented in Core Data. Every case that we have today is supported.

Furthermore, a pay-it-forward style of evolution is in place. When testing migration from version A to B, the file EntitySourceVC.swift must be created. The file graphC.json should be created even though it will not be used immediately. In this way, testing migration from B to C will already have the starting point of B defined and ready to go. This alleviates the need to remember (or derive) what was new in B in order to be able to test migration to C.

Add Versioned Data Structures

There is a script called gen-test (generate test) in the folder Banno/MigrationTests that produces Swift code from two managed object model versions. The two versions should be sequential as that is what is intended from the iterative migration architecture we use.

For the purposes of example, assume that the newly added managed object model version is 10, making the previous version 9. The script would be run as follows:

cd Banno/MigrationTests
./gen-test \
  --current ../Banno/Banno.xcdatamodeld/Banno\ 10.xcdatamodel \
  --previous ../Banno/Banno.xcdatamodeld/Banno\ 9.xcdatamodel

This will produce two files under Banno/MigrationTests:

  1. EntitySources/EntitySourceV010.swift containing the structs for version 10 of the MOM
  2. MigrationExpectations/MigrationExpectationsV010.swift containing the expectations of how data starting at version 9 gets transformed into version 10
  3. Migrate009to010Tests.swift containing the code that will initiate and validate the migration from version 9 to version 10

The newly generated files will be identified by Xcode automatically and made part of the MigrationTests target. Although there is no guarantee that these files are 100% ready to use after running the script, the script does handle all cases we have to date.

To be able to compile the newly generated code, it will be necessary to modify the EntitySourceVXXX.swift file for the previous managed object model version. Using the versions from the example above, this would mean opening EntitySourceV009.swift. At the bottom of the file, there would be a commented out type called EntitySourceContainerV009. That type must be uncommented so that the test code for migration to version 10 can use it.

NOTE: If an in-place managed object model change occurs, the script can be run again with the same two inputs. The existing Swift source files will be updated accordingly. Doing so will overwrite any customizations that were applied after the previous run of gen-tests. Those almost certainly must be re-applied. They may also be clues as to what was customized in the previous version of the mapping model.

IMPORTANT: There are limitations to the script that exist as of this writing (December 2024):

  • The generated migration expectations will not handle renaming of entities, attributes, or relationships.
  • The generated migration expectation assignment for new attributes currently only handles trivial cases: 0, false, or nil. If the managed object model has a default value set for an attribute, the generated code will handle applying the default value. For more complex cases, though, the Python script may require enhancement, or the generated Swift code may have to be altered.
  • The generated migration expectations will not do anything with custom transformation of data.

It is presumed that all of the above could be addressed by incorporating the heavyweight mapping model as input to gen-test. Current use cases have not required going to that next level, however.

Add/Update/Remove JSON Object Graph Entries

Going along with the pay-it-forward design approach, a JSON object graph file should be added for the new version. In this example, that would mean adding the file graphC.json under Banno/MigrationTests/ObjectGraphs. This should include all additions, modifications, and removals that correspond with the new MOM version. This file will serve as the starting point for the next test case (migration from C to D). Doing this now makes things easier in the future because all the knowledge about what has changed can be captured immediately rather than having to be pieced together at some later date.

The easiest way to make this new file is to do the following:

  1. Make a copy of graphB.json and call it graphC.json.
  2. Apply the relevant changes to align with MOM version C. This means adding new objects for new entities. Try to put some variety into new objects so that a teammate in the future can use graphC.json immediately when testing migration from C to D.
  3. Add graphC.json to the Xcode project.

Once everything compiles, try running the new test case.

Tips

Handling API Enumerations

Handling a Banno Consumer API enumeration requires extra care. The enumeration values from the API have, to date, always been strings. These can map to a Swift enum type that uses raw string values for the cases. Such a type cannot be used directly by Objective-C, but taking a Swift-first approach will pay dividends in the near term and in the long.

For the purposes of this example, let us assume that we want to add a new Core Data entity attribute for a corresponding JSON attribute called direction. The basic process for adding an enumerated attribute to an entity goes like this:

  1. Add the new attribute with String as the attribute type. For Core Data, the attribute name will be rawDirection. If there should be a default value, be sure to apply it when configuring the new attribute. If there is no reasonable default value, be sure to use String? for the type of the corresponding @NSManaged property in the NSManagedObject subclass.
  2. Add the Swift enum called Direction. (The code will be shown below.)
  3. In the NSManagedObject subclass for the entity, add two properties: rawDirection (for the actual Core Data detail) and direction. The latter will be a computed property that transforms from String to Direction.

For the Swift code, we will start with the Direction type that provides the client-side representation of the API enumeration.

enum Direction: String, CaseIterable, Equatable, Hashable {

    // The raw values must match the API values. In this case, deliberately
    // frustrating raw values have been chosen to demonstrate how we can
    // make client-side adjustments to align better with Swift conventions.

    case north = "DIRECITON_NORTH"
    case south = "DIRECITON_SOUTH"
    case east = "DIRECITON_EAST"
    case west = "DIRECITON_WEST"

}

Note that this does not conform to Decodable. It could conform to Encodable if we need to send values back to the Banno Consumer API. The reason for not conforming to Decodable is provided below.

Next, we will have a type called Example that provides the client-side implementation of the JSON interface contract. This will conform to Decodable because it will be used as part of decoding JSON received from the API.

struct Example: Decodable, Equatable {

    /// The raw direction string from the Banno Consumer API.
    ///
    /// Use of the raw value here avoids decoding problems that would
    /// occur if some new direction were added in the future. For example,
    /// if the API changed to include `"DIRECTION_SXSW"`, client-side
    /// decoding will keep working. The raw value will go into Core Data,
    /// but the `Direction` type will not be able to instantiate from that
    /// raw value until it is updated to include the new case.
    let direction: String

    /// Some other value for this data type.
    let value: Int

}

Then, we need the NSManagedObject subclass for the Example entity from the managed object model. We will say that the Example entity has been declared to have the attributes rawDirection (a String with no default value) and value (an Integer32 with a default value of 0). These correspond to direction and value from the Example type shown above.

final class ManagedExample: NSManagedObject {
}

// MARK: Fetch interface

extension ManagedExample: FetchableManagedObject {
}

// MARK: Attributes

extension ManagedExample {

    /// The "raw" value for the direction detail.
    ///
    /// The type here is `String?` because there is no default value in
    /// the managed object model applied at the time of inserting a new
    /// `Example` entity. If there were a default value, then the type
    /// should be `String`.
    ///
    /// For the possible values, see ``Direction``. For a more convenient
    /// interface, see the ``direction`` computed property.
    @NSManaged var rawDirection: String?

    /// Included to help with extrapolating beyond the enumerated value
    /// concepts.
    ///
    /// The backing attribute has a default value in the managed object
    /// model.
    @NSManaged var value: Int32

}

// MARK: Convenience accessors

extension ManagedExample {

    /// Transforms back and forth between the "raw" representation in
    /// ``rawDirection`` and the more convenient ``Direction``
    /// representation.
    ///
    /// If the value of `rawDirection` is an unsupported string, then `nil`
    /// will be returned. Calling code should treat `nil` as the unknown
    /// (or unsupported) case.
    var direction: Direction? {
        get {
            guard let rawDirection else { return nil }
            return Direction(rawValue: rawDirection)
        }

        set {
            rawDirection = newValue?.rawValue
        }
    }

}

The direction computed property provides the primary interface for application-level code. When that computed property returns nil, then appropriate logic should be applied to handle some case as being unknown or otherwise unsupported. Basically, Swift gives us the unknown/unsupported case automatically.

Finally, an extension on Example will define how an instance of Example is serialized to Core Data using an instance of ManagedExample.

extension Example: CoreDataSerializable {

    /// Required by CoreDataSerializable.
    func serialize(
        to managedExample: ManagedExample,
        in context: NSManagedObjectContext,
        index: Int,
        didCreate: Bool
    ) {
        // The raw direction value gets stored directly.
        managedExample.rawDirection = direction
        managedExample.value = value
    }

}

With all of that in place, bear in mind that NSPredicate and NSSortDescriptor use associated with Example entities will have to be in terms of the rawDirection attribute. Core Data and SQLite know nothing about the computed direction property. Objective-C compatibility can be added if necessary. Doing so depends on the circumstance and is beyond the scope of this documentation.

Custom Attribute Migrations

Sometimes the migration of an attribute value on some entity is not as simple as filling in a default or accounting for attribute renaming. When this arises, code in the file NSMigrationManager+MappingHelpers.swift typically comes into play. Adding an instance method provides a means to write custom code to transform a value from an entity using the previous version of the MOM to what is needed for the new version.

The custom migration method implementation can do just about anything that is needed. Bear in mind, however, that input will be as some instance of NSObject such as NSString or NSNumber. At this level, think of it more like KVC-style programming than consumption of a custom subclass of NSManagedObject. The Managed* classes cannot be used at all in this custom code.

Calling the custom migration method from the mapping model is not easy. A special expression has to be written in the model for the entity migration of the affected attribute. For example, V014 of the MOM alters the type of the depositExpirationDate attribute on PendingExternalTransferAccount. In the mapping model, the transformation of depositExpirationDate in PendingExternalTransferAccountTo PendingExternalTransferAccount needs the following expression:

FUNCTION($manager, "dateFromNumericRepresentation:" , $source.depositExpirationDateHolder)

This will cause NSMigrationManager.date(from:) to be invoked. The input will be the NSNumber representation of depositExpirationDateHolder from V013 (meaning a boxed int32_t value). The output will be the transformed Date representation. The NSMigrationManager instance will call the method with the correct input and handle assigning the result to the depositExpirationDate property on the transformed V013 entity.

Custom Relationship Migrations

When using a custom function for a relationship migration, it is necessary to tell the mapping model editor that the source is a custom expression. (This pertains when using Xcode 15.0 through 15.2 and possibly newer versions.) This setting is accessed through the relationship mapping inspector. The default is Auto Generate Value Expression. This must be changed to Use Custom Value Expression. Once that change is made, then it is possible to edit the value expression and have the modification “stick”.

Changing a One-to-One Relationship to be One-To-Many

IMPORTANT: See the information above about custom relationship migrations before reading this section.

If an existing one-to-one relationship needs to become a one-to-many relationship, the good news is that the default behavior with heavyweight mapping handles this automatically. The bad news is that a lot of production code may have to be adjusted to account for the property changing from a singular, likely optional type to a set of objects.

NOTE: The many-to-one form of the relationship will be represented as a set and will always be non-nil. What worked as a nil check in a one-to-one relationship needs to be updated to check for the number of items in the set instead.

To make the best use of the automatic migration through the heavyweight mapping model, make sure that the entity mapping for the one-to-many case appears in the list before the many-to-one case. For example, V018 of the MOM changed the account relationship on CreditCard from one-to-one to one-to-many. In the Account entity, the creditCard relationship was renamed to be creditCards. In the mapping model from V017 to V018, having CreditCardToCreditCard occur before AccountToAccount ensures that the existing relationship from CreditCard to Account is preserved.

In the persistent store migration test suite, a manual change can be applied after running the gen-test script. The following continues with the aforementioned example of V018 where the account relationship on CreditCard was changed to be one-to-many. In the generated file MigrationExpectationsV018.swift, the expectation for a migrated Account entity can be altered from this:

extension AccountV018: CompositeExpectation {
    init(_ source: AccountV018) {
        coreDataRelationships = source.coreDataRelationships
        // Remaining expectation setup
    }

to this:

extension AccountV018: CompositeExpectation {
    init(_ source: AccountV018) {
        coreDataRelationships = source.coreDataRelationships?.map { relationshipDescription in
            if relationshipDescription.name == "creditCard" {
                return RelationshipDescription(name: "creditCards", target: relationshipDescription.target)
            }
            return relationshipDescription
        }
        // Remaining expectation setup
    }

Doing so is a way to have the test suite confirm that the migration preserved the existing relationship between Account and CreditCard as a secondary check. The migration expectation for CreditCardV018 also confirms the preservation of the relationship. Having both checks mainly serves to add peace of mind. Of course, all of this only validates correctness if the input object graph results in a relationship being established between some CreditCard entity and an Account entity.