NopCommerce 4.3 (BETA) - Fluent Migrator workflow

Hello,

I downloaded the 4.3 beta code and have been looking at the code, there is a departure from EF to use Linq2DB and FluentMigrator.

As someone fairly new to FluentMigrator, what should the workflow look like if we are planning to do some custom development to the CORE NOPCommerce project to fit our needs? These changes will be part of the core functionality, not plugins.

For example, I noticed there is a class called SchemaMigration in Nop.Data, this class has an interesting attribute:

[NopMigration("2020/01/31 11:24:16:2551771", "Nop.Data base schema")]

The precision of the timestamp lead me to believe some sort of tool was used to generate this class, if that is the case, how is it generated?  

If I wanted to add a new entity, what is the best approach for creating this migration? I am used to using EF Core and the dotnet tools with commands such as:

dotnet ef migrations add "NewMigration"

Thank you!

Good question. Can we get some clarification on tekguy82's question?

I'm also keen on learning some more details about this switch from EF to linq2db.
The new migration way etc.

So this is just my experience with FluentMigrator while doing plugin development way before Nop added the support for it in 4.3. I hope Nop team will add more document related to this but so far I have not seen anything yet.

So to those who are not familiar with FluentMigrator, here is the quick run down on it. FluentMigrator is a framework that allows you to do code-first database migration instead of running manual update on database or SQL statements. In a scene, it is similar to EF Core migration feature but with some enhancements. You can read the comparisons online if you search for them.

Let's take a look at an example, say we want to add a new entity (new table). Normally with SQL script, you would have to write the CREATE TABLE statement and add the DB Context for the new table. Now with FluentMigrator, you just need to write a C# class that inherit FluentMigrator Migrations class.

[Migration(1, "Add NEW_TABLE table")]
[Tags("MyPluginTag")]
public class AddNewTableMigration : Migration
{
    public override void Up()
    {
        Create.Table("NEW_TABLE")
            .WithColumn("Id").AsInt32().PrimaryKey().Identity()
            .WithColumn("Name").AsString().NotNullable();
    }
}

So how will this code be executed to create a new table? In order for this execute, FluentMigrator runner would need be called in Nop so that it can scan for the assembly that contains those migration code and then be translate to SQL statement to execute within the DB for you.

Before 4.3, in order to get FluentMigrator working, you would have to either modify the Nop .Net Core pipeline to add the configuration for the framework or you can inject that through the plugin if you do not want to modify the Nop source code. I chose the latter since my work prefers not touching much of the Nop stuff. To get it working, I did the configuration in the start up process of my plugin (INopStartup class). Basically I have a method to generate a .Net core service that will return the FluentMigrator service. During the start up configure pipeline, I call that method and scope service so it will return the migrator runner. That will allow me to run up or down all the migrations.

In Nop 4.3, they added FluentMigrator directly into Nop configuration pipeline so no need to additional configuration (but you probably still want to do that and I can explain it later). Previously after creating new entity in the DB, you will have to create DB context for that new entity (entity class, mapping class, add the mapping class to the context, add dependency injection for the entity repository from the context, etc...). Now with the switch to linq2db, you don't have to do much as most of that works are being taken care behind the scene in Nop data layer. You can see some examples of that in some of the updated default plugins. However, you can still do some customization for it as you like. For example, they introduce INameCompatibility interface so that you can customize the table and the column names for the entity. Let say you have entity name MyNewEntity. By default, Nop will map that to the table with the name of MyNewEntity. But if your table name is different (i.e MyEntity), you will need to use the interface to specify that. Same thing for column name. Now if you decide to do a specific mapping configuration and not just globally, then you can still do that via the mapping class. Instead of inherit from NopEntityTypeConfiguration<T>, use the new NopEntityBuilder<T> and override MapEntity method where T is your entity class. Everything besides that are mostly the same with the repository pattern for accessing data in the DB.

Now my final point on this where I mentioned why you should customize the pipeline for migration instead of relying with the default provided configuration. If you look at some examples in the Nop plugin source code, you can see they are using NopMigration attribute and AutoReversingMigration. NopMigration attribute is the attribute that tells Nop this is a migration for NopCommerce and it should be picked up by the default runner. AutoReversingMigration is one of the built-in migration type of FluentMigrator that if there is something wrong during the up migration process, the runner will have to try to reverse the migration by running the down process instead. But why did I suggest to customize the migration process? The reason is that all of your migration will be keep tracked by Nop in a migration table that Nop defined initially. That means if there is something wrong with Nop migrations later on (update to a new Nop version), your migration could be affect and vice versa. Currently, FluentMigrator has a limitation that it can only apply migration down based on the migration number (the number that you specify in NopMigration attribute or Migration attribute) and not the order that the migration has been applied (based on time). So to avoid this pitfall, I decided to implement my own custom process with FluentMigrator when running the migrations from my plugin. That allowed me have my plugin migrations independent from Nop.

Here is the code that I used:

public static IServiceProvider CreatePluginMigratorService()
{
    var dataSettings = DataSettingsManager.LoadSettings();

    return new ServiceCollection()
        // Add common FluentMigrator services
        .AddFluentMigratorCore()
        .ConfigureRunner(rb => rb
            // Add SQLite support to FluentMigrator
            .AddSqlServer()
            // Set the connection string
            .WithGlobalConnectionString(dataSettings.ConnectionString)
            // Define the assembly containing the migrations
            .WithVersionTable(new CustomMigrationTable())
            .ScanIn(typeof(MyPlugin.AddNewTableMigration).Assembly)
            .For.Migrations()
            .For.VersionTableMetaData())
        // Enable logging to console in the FluentMigrator way
        .AddLogging(lb => lb.AddEventSourceLogger())
        .Configure<RunnerOptions>(opt =>
        {
            // Only run the migration if the tag is empty or equivalent to MyPluginTag
            opt.Tags = new[] { "", "MyPluginTag" };
        })
        // Build the service provider
        .BuildServiceProvider(false);
}

private void ApplyMigration(IApplicationBuilder application)
{
    try
    {
        var migrationServiceProvider = CreatePluginMigratorService();
        using var serviceScope = migrationServiceProvider.CreateScope();
        var runner = serviceScope.ServiceProvider.GetRequiredService<IMigrationRunner>();
        try
        {
            runner.MigrateUp();
        }
        catch (MissingMigrationsException)
        {
            // ignored
        }
    }
    catch (Exception)
    {
        // ignored
    }
}

public void Configure(IApplicationBuilder application)
{
    ApplyMigration(application);
}

So far it is working really well with the new codebase of Nop. I wish Nop would provide a better and easy way to customize that instead in the near future.

Really good explanation, well done!
I can see that you have experience with FluentMigrator and few questions comes to my mind and as we do not have something official from the Nop team I will ask them here.
1. What about the convenience of the auto migration from the EF - i.e. Add-Migration command. Is there something similar here? Or some equivalent to this?
2. What will happen if you already have a migration, you have applied this migration and on some later stage of the development you change something in this migration (let’s say add new column) in th Up method? Will you be able to apply it again, or for every new thing I have to create new migration?

KrisPetkov wrote:

To answer your concern:
1. If you look at the code that I provided early, the ApplyMigration method is executed within the Configure method. This Configure method is in my plugin startup, which is hooked directly into Nop configuration pipeline so whenever the plugin first run, it will try to run the migration before anything else.
2. For code first migration, once a migration is already in the system, it is better to create another migration to change things instead of modifying the existing one. In order to modify the existing migration, you would have to remove the tracking information within the migration table and that behavior is not recommended. For development, you can manually alter the db after the migration has ran to reflect the modification of that migration, but for live deployment, you should just add a new one and use that migration instead. Let's say you make a mistake that create a column in the table with the type of integer instead of float. Instead of changing the type in the initial migration, just make a second migration to change that type instead. That way the plugin will run the first and the second one and that will fix the issue. That should be the behavior for existing system. However, if you just want to install the plugin and have it fresh out of the box to create that column with float type, then changing that migration could be better.

KrisPetkov wrote:

I recently learned about this, there is a fluent migrator dotnet tool, see documentation at https://fluentmigrator.github.io/articles/runners/dotnet-fm.html

I was trying to use it but I ran into the same issue as this other user: https://github.com/fluentmigrator/fluentmigrator/issues/1241

Guys, I have one more question related to the linq2db and FluentMigrator entanglement.
Are you familiar with FluentMigratorMetadataReader.cs and NameCompatibilityManager.cs classes and what is he main idea behind them?
It is somehow related to the table/column naming, migrations and linq2db entities matching but I cannot grasp the whole idea of this approach. And isn't this too heavy to constantly iterating some classes, propeties, names etc.

P.S. do you know some good comprehensive tutorial for linq2db because their documentation is really not enough to learn how everything works in depth and what are the options that you have

KrisPetkov wrote:

I have not looked into those things in more details. However, by far what I have seen, both are related to the way Nop data layer handles data mapping. Previously, in order to get the repository to working with new entity, you would have to include the entity class, the mapping class, then add the mapping class to the plugin object context, and then register the dependency for the repository. With this new release, Nop cuts down the steps of setting up mapping class, plugin object context, and the registration for dependency of the repository. The idea of FluentMigratorMetadataReader class is to scan for new schema migration/entities, and then automatically map that entity using linq2db instead (In the comment, it is said "LINQ To DB metadata reader for schema created by FluentMigrator"). At the same time, NameCompatibilityManager class is to help backward compatibility with the mapping and resolves naming issues during entity mapping. For example, your entity has a name of EntityName, but for some reason your data table has a name of EntityNames. So in order for Nop to map correctly, you need to use either INameCompatibility interface or NameCompatibilityManager functions to resolve that discrepancy. That also includes column name in the table as well.

A more detail example is this: Let's you want to add a new table. First we define the entity that inherits from BaseEntity (i.e NewEntity : BaseEntity). Then we are going to create the migration for it.

public class NewEntity : BaseEntity
{
    public string NewEntityName { get; set; }
    public StatusEnum Status { get; set; }
}

[Migration(1, "First migration")]
public class AddNewEntityMigration : Migration
{
    private const string TABLE_NAME = "EntityCollection";

    public override void Up()
    {
        Create.Table(TABLE_NAME)
            .WithColumn("Id").AsInt32().PrimaryKey().Identity()
            .WithColumn("Name").AsString()
            .WithColumn("Status").AsInt32();
    }

    public override void Down()
    {
        Delete.Table(TABLE_NAME);
    }
}

After doing that, you're done as you can use IRepository<NewEntity> right away without doing registration on the DI or db object context (since it has been removed).

However, if your entity information matches with what the migration creates, then that should just work right of the box. But in my example, notice that my table name and one of the column name for the migration are different that the actual entity class. This will cause conflict and requires you to do additional mapping. With the new Nop version, you need to use INameCompatibility like this:

public class PluginDataMappingName : INameCompatibility
{
    public Dictionary<Type, string> TableNames => new Dictionary<Type, string>
    {
        { typeof(NewEntity), "EntityCollection" } // 
    }; // Map entity type NewEntity to table EntityCollection

    public Dictionary<(Type, string), string> ColumnName => new Dictionary<(Type, string), string>()
    {
        { (typeof(NewEntity), "NewEntityName"), "Name" }
    }; // Map property NewEntityName of entity type NewEntity with the column name 'Name'
}

Now what about the enum StatusEnum that I have above? What if I want to map it to different type in the database? In order to do that, you would have to include a NopEntityBuilder mapping class similar to the previous NopEntityTypeConfiguration.

public class NewEntityMappingBuilder : NopEntityBuilder<NewEntity>
{
    public override void MapEntity(CreateTableExpressionBuilder table)
    {
        table.WithColumn(NameCompatibilityManager.GetColumnName(typeof(NewEntity), nameof(NewEntity.Status))).AsInt32(); // Map enum to database integer type
    }
}

If you want to dig into the Nop codebase to know how to manage those things, I suggest start with MigrationManager.cs and walk backward.

As for linq2db document, yes that is so minimal but I believe you probably do not need to dig into that to develop your plugin since Nop abstract those layers within the codebase. So basically what you have done with previous version of Nop for repository pattern should just work the same, only slight different on the configuration part.

Thanks spincel.

This thread will help lot of plugin developers who are struggling for the new implementation of  data layer  by nop team.

We have struggled a lot to upgrade our plugins after implementation of this.

I would recommend  to contribute in documentation about Fluent Migrator workflow with nopCommerce in nopCommerce official documentation to help other developers who are looking in nopCommerce official documentation.

Thanks once again....