Saturday, May 7, 2016

Entity Framework 6 - Deploy migrations to staging server without breaking production server

tl;dr
Disable database initializer and use DbMigrator instead. See code snippet below or gist.

So for any given ASP.NET web app, we typically have this setup hosted on Azure:
  1. Dev web server / dev DB
  2. Automatically deployed to by master branch. Tested by developers.
  3. Staging web server / production DB
    Manually deployed to by human. Briefly test the new code against real production data before we switch all our users over to the new code.
  4. Production web server / production DB
    Manually deployed to by human. This is where all the traffic go, has multiple Azure web app instances to scale out.
Now, I want to add a new property to one of my entity classes and deploy that to production. Should be easy, right?

  1. Add a property to my Person class
  2. Run `add-migration Add-Person-MyNewProp` to scaffold a migration script
  3. Commit and push
  4. TeamCity builds the app, runs tests and creates a release in Octopus Deploy
  5. Octopus deploys the web app to the dev server
  6. Dev server reboots and migrates the dev DB on startup

"Hold on a minute, it migrates on startup? Mine doesn't?"
(Or let's pretend that is what you were thinking.)

Yes, isn't that sweet? I added the following to my Application_Start() method:

Database.SetInitializer<ApplicationDbContext>(
    new MigrateDatabaseToLatestVersion<ApplicationDbContext, Configuration>());

Ok, moving on. The new app is out in the cloud, you play with its test data and it's all good and ready to rock. Let's put it to the test on real production data, surely that will uncover some edge cases.

  1. Promote release from dev to staging in Octopus
  2. Octopus spins up the staging instance and deploys the exact same binaries
  3. Staging server reboots and migrates the production DB on startup
Awesome, this is the final test before the switch to production. You play around with the new stuff, it's all against real data and it kicks ass. Surely this release will go without a hitch. You go get the manager to show off the new cool feature, but he is busy in the phone and does not look happy.
Customers are calling about the server being down. You rush back to your [cubicle|dorm|ball pit], your inbox is flooded with system monitoring alerts. What is going on?!

Enough of the drama. The production server stopped working when our staging server migrated the production DB schema. The default behavior of Entity Framework code first is to fail - and fail hard - if the DB schema does not match the code. I can appreciate the safety check, but this was at least for me totally unexpected, since adding a new column should be backwards compatible.

It turns out, all of the database initializers that come bundled with Entity Framework, require the DB schema to match the code. So how can we then run a staging server against the production DB? 
  1. Disable the default database initializer, which defaults to CreateDatabaseIfNotExists
  2. Use DbMigrator to programmatically migrate the DB
I don't know what is the official approach, and maybe this whole ordeal is way off, but at least this works for me!


// Call me on startup, such as in Application_Start() of Global.asax.cs
private static void CreateOrMigrateDatabase()
{
    // Disable automatic migrations as it prevents us from deploying DB changes without breaking the production web instance.
    Database.SetInitializer<ApplicationDbContext>(null);

    var migrator = new DbMigrator(new Configuration());
    
    // Order by migration names to get oldest first, such as 201403221133523_AddSetting
    List<string> pendingMigrations = migrator.GetPendingMigrations().OrderBy(x => x).ToList();
    List<string> localMigrations = migrator.GetLocalMigrations().OrderBy(x => x).ToList();
    List<string> databaseMigrations = migrator.GetDatabaseMigrations().OrderBy(x => x).ToList();

    // Sanity check, avoid trying to migrate a DB that is newer or has different history than our local migrations
    if (pendingMigrations.Any())
    {
        bool migrationsCompatibleWithDb = localMigrations.StartsWith(databaseMigrations);
        if (!migrationsCompatibleWithDb)
        {
            var ex = new Exception("Unable to migrate database. Not compatible.");
            ex.Data["localMigrations"] = string.Join("\n", localMigrations);
            ex.Data["databaseMigrations"] = string.Join("\n", databaseMigrations);
            ex.Data["pendingMigrations"] = string.Join("\n", pendingMigrations);
            throw ex;
        }

        Trace.TraceInformation($"Migrating DB from {databaseMigrations.LastOrDefault() ?? "(empty)"} to {pendingMigrations.Last()}");
        migrator.Update(pendingMigrations.Last());
    }

    using (var db = new ApplicationDbContext())
    {
        db.Database.CreateIfNotExists();
        db.Database.Initialize(false);
    }
}


And for my own convenience, I use this extension method:

using System;
using System.Collections.Generic;
using System.Linq;

namespace MyApp.Extensions
{
    public static class EnumerableExtensions
    {
        public static bool StartsWith<T>(this IEnumerable<T> source, IEnumerable<T> items)
        {
            if (source == null) throw new ArgumentNullException(nameof(source));
            if (items == null) throw new ArgumentNullException(nameof(items));

            IEnumerator<T> sourceEnumer = source.GetEnumerator();

            // False if source is shorter than items or any source item
            // does not equal the item at the same index.
            return items.All(item => sourceEnumer.MoveNext() && Equals(sourceEnumer.Current, item));
        }
    }
}

Saturday, February 6, 2016

Automating your business or personal life in 2016

I've always had a warm fuzzy feeling when it comes to automating things and in the recent years I've seen a new type of developer service gaining traction that makes it really ease to automate stuff.

So this was me a few years ago
Hey, I can automate that with this piece of code. It's so easy!
Then as the years went and I grew a beard
Do we really need to write code for this? Why must things be so hard!
Fortunately for us grumps, we no longer have to!

Scott Hanselman was the one to open my eyes to this world. In an excellent video he demonstrates the real life case of an automobile repair shop that wanted to send out an SMS when the car was ready for pickup. They were running a really old system and hiring coders to develop this custom integration would be quite costly to create and maintain.

With a few clicks and drag'n'drop, Scott hooked up an Azure logic app to monitor their SQL database for new rows, and when a Status column changed value to "Done" it would automatically grab the contact info from the same row and send out an SMS to the customer with a personalized message using the customer's name and car model. Not a single code involved, no redeployments of services and immediate business value added to their shop. This is great!

I couldn't find the original video, but here is a similar one.

Automation services


Zapier

supports an impressive 500+ APIs out of the box and has all the popular services you can think of like Facebook, Twitter, Google Drive, Office 365. They recently got support for the much anticipated multi-step and supports scheduling with relative or absolute delays between actions. Their UI used to be real clunky and awkward, but they made great improvements lately.

I still find it disappointing they have a hard limit of scheduling actions more than 1 month into time, which makes it harder to follow up on things like 12 month warranties on purchased products, but with Mandrill's built-in scheduling I can work around that for at least emailing.

Azure logic apps 

is pretty new, but started out great with multi-steps built in from the start. They still don't support scheduling or delays, but I can only imagine this will be added soon. Their API integrations are very limited compared to Zapier, but has great support for Microsoft services and do support generic HTTP request integrations so you can talk to just about any HTTP API with a bit of customization.

IFTT (If This Then That)

focuses on the end user and allow users to share automation recipes for automating your personal life. Examples include posting to Slack when you enter the office in the morning or muting the phone at bedtime.

I'm missing some business oriented features, such as custom integrations with my own apps and APIs multi-step recipes and scheduling. According to their help page custom integrations are being developed.

Amazon AWS Data Pipeline 

is a service I haven't personally tested yet, as I don't use Amazon much. From their landing page it seems to focus on automating workflows between your services and manipulating data.


So what are you waiting for? Start automating your business and/or life.

Sunday, January 31, 2016

IIS - Exposing localhost to the world for free

There are many ways to expose your localhost, such as the free option localtunnel.me and the paid option ngrok.com.

Localtunnel gives you a random URL that only lives as long as your connection is held open, which can be a pain when you have to reconfigure external services to call your app every now and then.

Ngrok allows you to reserve a fixed subdomain, but starts at $60/year and $21/month (minimum 3 users for monthly, yuck!).

So, what is a dirt cheap coder to do? We roll our own of course.

Assumptions:
  1. You already have IIS Manager installed, if not you know what to do. It's free!
  2. You are behind a fixed IP or at least it doesn't change too often.
  3. You already own a mydomain.com and can point app.mydomain.com to your IP. I highly recommend cloudflare.com for managing your DNS records. It's free and awesome.
  4. Your local app is running on localhost:8000

IIS as Reverse Proxy

There are many tutorials on this, so I'll list the main points.
  1. Add a website and point it to some empty folder with the proper permission.

    If your website's application pool is named app.mydomain.com then you want to add read/list/execute permissions to the Windows user IIS AppPool\app.mydomain.com.
     
  2. Go to URL Rewrite of your website, then add a Reverse Proxy rule. Fill in localhost:8000 and choose to rewrite the HTTP links.

     
  3. Profit! No? You are probably seeing an IIS error about not being able to forward gzip'ed responses from the local server. Find the web.config of your reverse proxy website folder (no not your app's web.config) and add the bold sections below. This will strip the Accept-Encoding header on incoming, and add it back on outgoing so the local server does not think the caller accepts gzip'ed HTTP. Sweet.

    The original stackoverflow post on reverse proxy and gzip.
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="ReverseProxyInboundRule1" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="http://localhost:8000/{R:1}" />
                    <serverVariables>
                        <set name="HTTP_X_ORIGINAL_ACCEPT_ENCODING" value="{HTTP_ACCEPT_ENCODING}" />
                        <set name="HTTP_ACCEPT_ENCODING" value="" />
                    </serverVariables>
                </rule>
            </rules>
            <outboundRules>
                <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsHtml1">
                    <match filterByTags="A, Form, Img" pattern="^http(s)?://localhost:8000/(.*)" />
                    <action type="Rewrite" value="http{R:1}://app.mydomain.com/{R:2}" />
                </rule>
                <rule name="RestoreAcceptEncoding" preCondition="NeedsRestoringAcceptEncoding">
                  <match serverVariable="HTTP_ACCEPT_ENCODING" pattern="^(.*)" />
                  <action type="Rewrite" value="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" />
                </rule>
                <preConditions>
                    <preCondition name="ResponseIsHtml1">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                    </preCondition>
                    <preCondition name="NeedsRestoringAcceptEncoding">
                      <add input="{HTTP_X_ORIGINAL_ACCEPT_ENCODING}" pattern=".+" />
                    </preCondition>
                </preConditions>
            </outboundRules>
        </rewrite>
    </system.webServer>
</configuration>


Now, either point your domain to your IP or simply edit your local hosts file to point it to 127.0.0.1, and access app.mydomain.com from your browser and watch your local server present its glory stuff.