MongoDB is well known for not having “schema” but this is a bit of a misnomer. The more appropriate way to describe it would be, MongoDB does not enforce a schema. If you take the literal definition of schema, you get:

a representation of a plan or theory in the form of an outline or model.

MongoDB does not require you to know what the data will look like before inserting it into the database. So in this sense, MongoDB does not, in fact, have schema. However, our programs that interact with MongoDB do have a schema. We insert data into the database in a particular way, and we expect it back in that way. Perhaps our schema is extremely flexible and we have many different forms of documents, but at the end of the day, there is a schema as defined by the software we write.

One of the biggest mistakes somebody can make when dealing with a database that doesn’t enforce schema, is to put unrelated items in a collection. While the database allows this, it can make managing the data quite hard. This is a very good case of “just because you can, doesn’t mean you should.” Lets imagine for a minute that we are implementing a blog on top of MongoDB (This is an example used in the MongoDB Docs). There is nothing stopping us from putting the blog posts, the comments, and the users all in one monolithic collection. It just makes our code very ugly. We need a way to delineate posts from comments from users, which might merit the addition of a type field, or perhaps we filter client side based on the existence of some fields.

If we were to model this in typescript, it might look something like this:

1
2
3
4
5
6

// Comment Schema
{ '_id': ObjectId, 'postId': ObjectId, 'comment': string, 'userId': ObjectId, 'type': string }
// Post Schema
{ '_id': ObjectId, 'authorId': ObjectId, 'content': string, 'tags':string[], 'date': Instant, 'type': string }
// User Schema
{ '_id': ObjectId, 'name': string, 'pwd': string, 'email': string, 'type': string }

Some common lookups would look something like

1
2
3
4

// All posts
db.allObjects.find({'type': "post"});
// All comments for a post
db.allObjects.find({'type': "comment", 'postId': <ObjectId>});

As you can see, there is no advantage to doing this over

1
2
3
4

// All posts
db.posts.find();
// All comments for a post
db.comments.find({'postId': <ObjectId>});

The latter is considerably more self explanatory than the former. This makes the code considerably more readable. It also makes it much easier to do in C#. If we translate the above 2 examples to C#, we can see why quite easily.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

public class Comment
{
    public ObjectId Id { get; set; }
    public ObjectId PostId { get; set; }
    public string Comment { get; set; }
    public ObjectId UserId { get; set; }
    public string Type { get; set; }
}
public class Post
{
    public ObjectId Id { get; set; }
    public ObjectId AuthorId { get; set; }
    public string Content { get; set; }
    public string[] Tags { get; set; }
    public Instant Date { get; set; }
    public string Type { get; set; }
}
public class User
{
    public ObjectId Id { get; set; }
    public string Name { get; set; }
    public string Pwd { get; set; }
    public string Email { get; set; }
    public string Type { get; set; }
}

Now to retrieve those documents

1
2

await db.GetCollection<Post>("allObjects").Find(Builders<Post>.Filter.Eq(x => x.Type, "post")).ToListAsync();
await db.GetCollection<Comment>("allObjects").Find(Builders<Comment>.Filter.Eq(x => x.Type, "comment") & Builders<Comment>.Filter.Eq(x => x.PostId, <ObjectId>)).ToListAsync();

As you can see, there is no advantage to doing this over

1
2

await db.GetCollection<Post>("posts").Find(Builders<Post>.Filter.Empty).ToListAsync();
await db.GetCollection<Comment>("comments").Find(Builders<Comment>.Filter.Eq(x => x.PostId, <ObjectId>)).ToListAsync();

The big danger with doing a single collection in C# with MongoDB is if you accidentally do the following:

1

await db.GetCollection<Post>("allObjects").Find(Builders<Post>.Filter.Eq(x => x.Type, "comment")).ToListAsync();

So now that we all agree, while MongoDB does not enforce schema, there is a schema that must be followed for the sanity of all those who interact with the database. With this in mind, lets look at some ways of modifying that schema without causing downtime.

Performing Migrations

While I am going to talk a lot about performing migrations in C#, the methodology I discuss is applicable to most, if not all, languages that interface with MongoDB.

The C# driver team has written some functionality into the C# driver for handling these situations (docs. I don’t generally use this method, mainly because I do not like the idea of documents sitting in an “inconsistent” state. I prefer the method of updating all the documents as quickly as possible. Now there are a few types of schema changes we need to address (as laid out in the driver docs):

1. A new member is added
2. A member is removed
3. A member is renamed
4. The type of a member is changed
5. The representation of a member has changed

A New Member is Added

When a new member is added, there are a few options to work with

• Use a default value in the POCO (probably best for value types)
• Handle null (or default) values in code (probably best for reference types)
• Update all documents in the database with the appropriate value

Default Value on the POCO

1
2
3
4
5
6
7
8

public class ExistingDocumentWithNewValueField
{
    public ObjectId Id { get; set; }
    public string Foo { get; set; }
    public string Bar { get; set; }
    // This is our new field
    public bool IsFooBar { get; set; } = true;
}

Now the default for IsFooBar is not default(bool) and we can happily continue on our way. This works great for anything that is easily defaulted, like value types.

Handle Null Values in Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

public class ExistingDocumentWithNewReferenceField
{
    public ObjectId Id { get; set; }
    public string Foo { get; set; }
    public string Bar { get; set; }
    // This is our new field
    public FooBar FooBar { get; set; }
}
public class FooBar
{
    public string Baz { get; set; }
    public int Bob { get; set; }
}

The default for FooBar is going to be null on deserialization. When we access this member, we must do a null check, which, depending on our code, might be a valid state for FooBar. While it is possible to use the first example in this case as well, generally classes are harder to construct in a default way as they are generally driven by some other piece of data not already existing in the current document.

Update all Existing Documents with the Appropriate Value

In this example, I am simplifying the value of FooBar to be something simple, however I expect in the real world, the process of constructing FooBar will be more complex. Our update process looks something like:

1. Update the POCO to contain the field, but don’t yet use it in code
2. Update the database to add the new default value
3. Update the code to use the new field in the POCO

So re-using the classes from handling null values in code, our update would look something like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<ExistingDocumentWithNewReferenceField>("existingNewRef");
var docsCursor = await collection.Find(Builders<ExistingDocumentWithNewReferenceField>.Filter.Empty).ToCursorAsync();
var count = 0;
while(await docsCursor.MoveNextAsync())
{
    foreach(var doc in docsCursor.Current)
    {
        var fooBar = new FooBar() { Baz = count.ToString(), Bob = count };
        await collection.UpdateOneAsync(
            Builders<ExistingDocumentWithNewReferenceField>.Filter.Eq(x => x.Id, doc.Id),
            Builders<ExistingDocumentWithNewReferenceField>.Update.Set(x => x.FooBar, fooBar));
        count++;
    }
}

This isn’t the most efficient way to do the updates (using the bulk API would be faster), but it gets the job done. There is a small problem with this issue. We’ve added a new field to ExistingDocumentWithNewReferenceField, but this is live code, so unless we’ve decorated the ExistingDocumentWithNewReferenceField with [BsonIgnoreExtraElements], we’ll run into errors in our running code that don’t yet know about the new FooBar field. There are 2 ways around this

• The previously mentioned [BsonIgnoreExtraElements]
• Deploy a new version of our app that knows about FooBar but doesn’t yet do anything with it

I prefer the second one for no particular reason, but the first is just as valid. Each requires forethought before the migration, so the choice is up to you.

A Member is Removed

This case also requires a bit of forethought and a decision to:

• Leave the data in place but ignore it: (See docs: Supporting Extra Elements, Ignoring Extra Elements)
• Remove the data from all documents

I prefer to remove the field from the database instead of just ignoring it as it can cause some confusion when looking at the data manually. To remove the data in an online fashion:

1. Remove all references to the field in code, but leave the field in the POCO
2. Update the database to remove the field from all documents
3. Update the POCO to remove the field

The reason I opt for this method is that I can still use the strong typed driver interfaces to remove the field(s) and don’t need to worry about fat fingering the update(s). This method make use of the $unset operator in MongoDB to remove the field, not just set it to null. So a migration to remove our field would look like

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<ExistingDocumentWithNewReferenceField>("existingRemoveRef");
var docsCursor = await collection.Find(Builders<ExistingDocumentWithNewReferenceField>.Filter.Empty).ToCursorAsync();
while(await docsCursor.MoveNextAsync())
{
    foreach(var doc in docsCursor.Current)
    {
        await collection.UpdateOneAsync(
            Builders<ExistingDocumentWithNewReferenceField>.Filter.Eq(x => x.Id, doc.Id),
            Builders<ExistingDocumentWithNewReferenceField>.Update.Unset(x => x.FooBar));
    }
}

Once this update completes, we can update our code to remove the FooBar field from the POCO.

A Member is Renamed

Think of this as a combination of adding and removing a member.

1. Update the POCO to have the new member name
2. Update code references to prefer the new member name but default to the old member name
3. Update the database, moving the data from the old name to the new name
4. Update the POCO to remove the old member name

This update is can be very easy thanks to the $rename operator in MongoDB. Instead of having to query each document and updating them one by one, we can issue a single update statement.

First, lets look at our schema:

1
2
3
4
5
6
7
8
9

public class ExistingDocumentWithFieldRename
{
    public ObjectId Id { get; set; }
    public string Foo { get; set; }
    public string Bar { get; set; }
    public FooBar FooBar { get; set; }
    // This is our new field
    public FooBar BazBob { get; set; }
}

We are going to rename our FooBar field to BazBob, better describing the content.

1
2
3
4
5
6

var client = new MongoClient();
var db = client.GetDatabase("test");
var collection = db.GetCollection<ExistingDocumentWithFieldRename>("renameField");
await collection.UpdateManyAsync(
    Builders<ExistingDocumentWithFieldRename>.Filter.Exists(x => x.FooBar),
    Builders<ExistingDocumentWithFieldRename>.Update.Rename(x => x.FooBar, nameof(ExistingDocumentWithFieldRename.BazBob)));

Note that the Update.Rename method doesn’t take 2 field selectors, it takes a field selector and a string of the new name. To ensure we don’t fat finger this one either, we can make use of the nameof operator in C#. This update will apply to all documents in the collection, although it is not an atomic operation. The update on each document is atmoic, but the whole update process is not. This is why our code must be tolerant of either field existing.

The Type of a Member is Changed

This process is probably the most complex of all, depending on what the current and desired types are. If the types are implicitly convertible in the .NET Runtime, you can most likely just leave the data in the database alone. If the types are not convertible, this gets a bit complex. I think the safest and easiest way to handle this is to make use of both the Rename steps and the New Member is Added steps.

1. Update the POCO to have a new, temporary, field with the old type
2. Update the the application code to handle reading from either the old or new type (perhaps with an in-memory conversion of old -> new type)
3. Update the database to $rename the old field to the temporary field name
4. Update the POCO to change to the new type on the existing field
5. Update the database to convert from the old type (now in the temporary field) to the new type in the original field by reading the documents, converting the types in memory, and writing the converted value
6. Update the database to $unset the temporary field
7. Update the POCO to remove the temporary field

I have not personally had to perform this type of schema migration, so there may be craftier, more efficient, ways to perform this, but I am unaware.

The Representation of a Member is Changed

The MongoDB C# driver chooses the serializer to use for reading BSON based on the BSON type, not the member type in the POCO. That conversion happens later. This means you can typically alter the representation of a type without altering the POCO so long as the .NET Runtime has a way to perform a conversion. If the runtime lacks this conversion, you will need to follow the procedure for when the type of a member has changed.

How to Implement These Conversions in Your Application

So now that we understand the different methods needed to actually alter our schema and underlying stored data. I have chosen to use a schema version document and migration scripts at startup. This means deploying new migrations looks something like:

1. Deploy a new app version to support any temporary or new fields
2. Deploy a version with the migrations
3. Deploy a new app version that removes any temporary or unneeded fields

Depending on the migration, steps 1 and 2 may be combined. For tracking the schema versions, I have a document in my database with the current schema version:

1
2
3
4

{
    'currentSchemaVersion': 22,
    'currentAppVersion': "1.0.3.243"
}

My web application is generally what needs to be tolerant of migration changes mid-migration. My background workers block attempting to acquire a migration lock on startup. My migration runner follows the steps below

1. Find the migration document, if it doesn’t exist, create it using an _id of the environment name (MongoDB won’t let multiple documents have the same _id).
2. Attempt to acquire a lock (using SharpLock, see Distributed Locks in C#)
3. Find all migrations in the codebase by the interface IMigration
4. Sort the migrations found by their version (lowest to highest)
5. Skip any migrations with a version less than the version in the migration document
6. Run the migrations in order
7. Release the lock

This means the first worker process to acquire the lock will run all the migrations serially. Once it is done, it will release the lock, any subsequent workers that acquire the lock will filter out all migrations that have run already and continue their startup.

The IMigration interface is quite simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

internal interface IMigration
{
    int GetMinSchemaVersion();

    /// <summary>
    /// Run the migration code
    /// </summary>
    /// <param name="servicesProvider">The <see cref="IServiceProvider">collection of services</see>.</param>
    /// <param name="migrationStatus">The <see cref="MigrationStatus"/> document. This is not guaranteed to be the most up to date reflection of this document.</param>
    /// <returns></returns>
    Task<bool> RunMigrationAsync(IServiceProvider servicesProvider, MigrationStatus migrationStatus);
}

Conclusion

Migrations in MongoDB using the C# driver don’t have to require you to drop to the command line. They can be done with type safety and with no downtime if implemented correctly.

Happy coding!