CD

Enforce Visual Studio Build Action in Projects – dotnet core Edition

Posted on by Sam

In a previous post I described a method for enforcing a certain msbuild BuildAction on a sub-set of the project’s files, generating a build error if the expected action was not set. I now wanted to use this in a dotnet core project and with a minor tweak, it works just the same.

The use case is still the same, I’m using the lovely DbUp to perform DB schema migrations during deployments as part of an autonomous CI/CD setup. A pull-request which was recently merged contained SQL files which were not flagged as EmbeddedResources, meaning they were not picked up and executed (lucky for them!), resulting in a deployment failure. To prevent this happening again, I wanted to add the same check in as before.

I began by checking to see whether everything I needed was supported in the dotnet core world and luckily, support for XmlPeek (and XmlPoke) was added earlier this year. Woot. I created a new branch, opened up the project file, copy/pasta the old target and build. No errors. Good! Next I changed one of the SQL files to have a BuildAction of None, build, no errors. Not good.

After some trial and error, I discovered that the dotnet core .csproj files no longer use the XML namespace (http://schemas.microsoft.com/developer/msbuild/2003) that their older siblings use. I couldn’t find a reason why, but the MS project migration documentation simple states to remove them. Fair enough.

The updated target for a dotnet core project is below. Simple replace anything ‘SQL’ related to suit your own needs.

<Target Name="EnsureSQLScriptsAreEmbeddedResource" BeforeTargets="CoreCompile">
  <XmlPeek XmlInputPath="$(MSBuildProjectFile)" Query="Project/ItemGroup/*[not(self::EmbeddedResource)]/@Include">
    <Output TaskParameter="Result" ItemName="AllItems" />
  </XmlPeek>
  <ItemGroup>
    <Filtered Include="@(AllItems)" Exclude="SqlTemplate.sql" Condition="'%(Extension)' == '.sql'" />
  </ItemGroup>
  <Error Code="SQL" ContinueOnError="ErrorAndContinue" File="$(MSBuildProjectDirectory)\%(Filtered.Identity)" Text="All scripts require a BuildAction of EmbeddedResource" Condition="'@(Filtered)'!=''" />
</Target>

Deploy an ASP.Net Core 2.0 Application to an Azure App Service With Bitbucket Pipelines

Posted on by Sam

Recently I took another look at Bitbucket’s Pipelines offering, their successor to Bamboo cloud which was removed last year (but lives on as an on-premise solution). Pipelines had previously been uninteresting to me as there was no support for traditional .NET apps, but now that I’m dealing more with .NET Core, I figured it would be worth a look. As it turns out, I abandoned this approach almost as soon as I had things up and running, choosing to go with the built in support offered by Azure App Services. A guide on that will appear soon.

The premise is cool, builds are run in Docker containers, defined using YAML and you get 50 build minutes included in the free account, with all the triggers and notifications you’d expect, e.g. allowing you to fire off Slack notifications to keep your team informed of the build health. If your code is sitting in a Bitbucket repo, it makes sense to try and keep everything under one roof (always with an alternative solution of course, you never know when that roof will collapse!).

I started off by following the guide provided by Atlassian and the YAML from this repository which immediately resulted in errors. Getting the build definition in to shape took a bit of messing around so I decided to write this post in the hopes that a) you can laugh at the silly mistakes I made along the way and b) provide a more up to date and comprehensive guide to deploying an application to the Azure App Service using Bitbucket Pipelines.

Trials and Errors

Problem #1 – Builds not being triggered

Cause: Incorrect casing of the branch name in the build definition file, ‘Development’ vs. ‘development’ (who would name the development branch ‘Development’ anyway?!).

Solution: Correct the casing in the bitbucket-pipelines.yml file.

Problem #2 – Error building the solution

+ dotnet build $PROJECT_NAME
Couldn't find 'project.json' in 'PROJECT_NAME'

Cause: This was the result of project.json being a dotnet core 1.0 feature which is now deprecated in version 2.0.

Solution: As this is a dotnet core 2.0 project, I needed to use a different Docker image, and changed the Docker image from

image: microsoft/dotnet:onbuild

to

image: microsoft/dotnet:2.0-sdk

Problem #3 – Error restoring Nuget packages

  + dotnet restore
  MSBUILD : error MSB1003: Specify a project or solution file. The current working directory does not contain a project or solution file.

Cause: It would appear that at some point between version 1 and 2, an error is raised instead of a warning when the restore command is unable to find a project/solution file. The repository was structured such that the solution was in a folder off the root directory.

Solution: Pass in the path to the solution file to the restore command. E.g.

- dotnet restore FolderWithSolution/MySolution.sln

Problem #4 – Error pushing to the Azure App Service Git repository

  + git push https://$AZURE_LOGIN:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net/appservicename.git master
  error: src refspec master does not match any.
  error: failed to push some refs to 'https://username:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net/appservicename.git'

This tripped me up as I hadn’t taken the time to process what the error was telling me and ended up on a bit of a wild goose chase, resulting in me cloning the App Services Git repository locally and creating a master branch. Note: Do not do this, jump to #6 to see why.

Cause: The build was being executed on the Development branch, but I was trying to push code to the App Services’ master branch.

Solution: Change the Git push command, specifying the source and destination branch names:

git push https://$AZURE_LOGIN:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net/appservicename.git Development:master

Problem #5 – Error pushing to the Azure App Service Git repository, take two

+ git push https://$AZURE_LOGIN:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net/appservicename.git Development:master
  fatal: unable to access 'https://username:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net/appservicename.git/': Illegal port number

Well that’s weird! HTTPS is pretty standard after all no? Just for good measure I went and changed the Git push command again, adding the port in:

git push https://$AZURE_LOGIN:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net:443/appservicename.git Development:master

Alas the error persisted.

Cause: Embarassingly, the value for the $AZURE_PASSWORD environment variable contained a colon. Oh dear, schoolboy error! That’s what you get for trying to generate a super secure password.

Solution: Change the Azure App Services deployment password such that it doesn’t contain any characters that will break the URL.

Problem #6 – Error pushing to the Azure App Service Git repository, take three

  + git push https://$AZURE_LOGIN:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net:443/appservicename.git Development:master
  To https://appservicename.scm.azurewebsites.net:443/appservicename.git
   ! [rejected]        Development -> master (fetch first)
  error: failed to push some refs to 'https://username:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net:443/appservicename.git'
  hint: Updates were rejected because the remote contains work that you do
  hint: not have locally. This is usually caused by another repository pushing
  hint: to the same ref. You may want to first integrate the remote changes
  hint: (e.g., 'git pull ...') before pushing again.
  hint: See the 'Note about fast-forwards' in 'git push --help' for details.

Cause: Sigh. In order to create the master branch in problem #4, I had to push a commit, which meant that the repository had changes that would need to be merged. Something felt very wrong here so I decided to reset the Azure App Service Git repository, which is easier said than done.

Solution: I ended up deleting and re-creating the App Serivce, using the same details as before so that I wouldn’t have to go through and update my build definition and environment variables.

Success

At last, I saw a nice green build indicator. Finally! I’d eaten through 13 of those precious 50 build minutes by this point, but it was working and the builds were relatively quick, taking around 3 minutes apiece. Here’s what the bitbucket-pipelines.yml looked like at this point:

image: microsoft/dotnet:2.0-sdk

pipelines:
  branches:
    Development:
      - step:
          caches:
            - dotnetcore
          script:
            - dotnet restore SolutionName/SolutionName.sln
            - dotnet build SolutionName/ProjectName
            - git push https://$AZURE_LOGIN:$AZURE_PASSWORD@appservicename.scm.azurewebsites.net:443/appservicename.git Development:master

Once the code was being deployed, I hit the next issue. This repository contained a solution with two ASP.NET Core web apps and I couldn’t easily figure out how to specify which project to deploy. Deploying the entire build output would always result in the first website being active. At this point, due to other checkins, all of those 50 build minutes were used up! Towards the end, when building two projects, the builds were hitting 9 minutes. Rather than pony up the $10 for another 1000 minutes, I took my search elsewhere.

The Recipe for Success

If you want to give Bitbucket Pipelines a spin and would like to avoid my mistakes, follow this guide. These steps assume three things:

  1. You already have your app code sitting in a Bitbucket repository.
  2. You have an active Azure subscription and have already created your App Service.
  3. Your .NET Core solution has a single project you wish to deploy.

This guide will bypass the Bitbucket Wizard for adding the bitbucket-pipelines.yml file to your repository as it only allows you to commit this directly to your master branch. If that’s ok with you, then follow the wizard (just access the Pipelines for your repository and follow the steps), however the steps below will result in the same outcome.

Please refer to the Atlassian documentation for the bitbucket-pipelines.yml file specification.

Note: Pipelines will only run for branches in which the bitbucket-pipelines.yml file exists and are defined in the build file (unless you only use a default configuration). Depending on your Git workflow, you may want to add this file to a development branch first, then propagate it to other branches by merging.

  1. Create a new text file, in the root of your repository on the desired branch, with the name bitbucket-pipelines.yml
  2. Paste in the following content: 
    image: microsoft/dotnet:2.0-sdk

pipelines:
  branches:
    BRANCH_NAME:
      - step:
          caches:
            - dotnetcore
          script:
            - dotnet restore SOLUTION_FOLDER/SOLUTION_NAME.sln
            - dotnet build SOLUTION_FOLDER/PROJECT_NAME
            - git push https://$AZURE_LOGIN:$AZURE_PASSWORD@APP_SERVICE_NAME.scm.azurewebsites.net:443/APP_SERVICE_NAME.git BRANCH_NAME:master
  3. Go through and replace the values for the following to match your setup:
    • BRANCH_NAME
    • SOLUTION_FOLDER
    • SOLUTION_NAME
    • PROJECT_NAME
    • APP_SERVICE_NAME
  4. If you don’t have multiple branches to deal with, you can omit the branches: and branch name (BRANCH_NAME:) lines, replacing them with default:
  5. Commit the file and push to your remote
  6. If you did everything right so far, you should be able to navigate to the Pipelines section of your Bitbucket repository, scroll down and see the contents of your bitbucket-pipelines.yml file in the validator. If there are any validation errors, correct them now
  7. Click the Enable button Enable Bitbucket Pipelines
  8. To define the username and password that will be used to push to your App Service’s Git repository, you have two options. Follow this guide to set those credentials.
  9. With the credentials in place, open the repository in Bitbucket and navigate to Settings -> Pipelines -> Environment Variables
  10. Add a new variable called AZURE_LOGIN and set the value to your deployment username. Mark this as secured if you like
  11. Add a second variable called AZURE_PASSWORD, setting the value to your deployment password. It’s recommended that you mark this as secured to prevent it appearing in logs
  12. Commit a change to the branch you added the bitbucket-pipelines.yml file to and you should be all set! If you hit any errors, check out the list of problems I ran in to and see if you’re experiencing the same. Otherwise, hit up Google, or ask a question in the comments below

Enforcing BuildAction on VS Project Items

Posted on by Sam

Whilst creating an application to run the excellent SQL migration tool DbUp as part of a continuous deployment setup, I stumbled across a question I hadn’t thought about before. Is it possible to enforce a particular build action on certain files in a Visual Studio project? In this particular case, making sure all SQL files are marked as EmbeddedResource.

With DbUp, there are several ways of supplying SQL files to be run during a migration. In my case, I don’t know which set of scripts need to be run until the deployment is underway. After toying around with several ideas, I settled on the approach of embedding the SQL files in to the migration tool. I can then use the connection string that’s passed in to determine a specific folder of scripts to be executed. Having the scripts as embedded resources will allow me to package up the migration tool as an artefact that can be run only when needed, i.e. during a deployment and not during a build.

So how does it work?

There are several ways of adding custom targets to a project file. After Googling around I came across the ever helpful StackOverflow which offered up some solutions. The first approach is simple, but would require you to add a case for each of the BuildAction types that exist. And if a new one is added at some point, you have to update your code. No thanks, I’m too lazy for that.

The second approach looked much more maintainable by focusing on everything that isn’t what you want. The problem was that it didn’t work! Nothing would ever be matched. I wasn’t able to see any information in Visual Studio, so ran the build from the command line which told me the target was being run, but no elements were matched.

After lots of trial and error and head scratching, I came across another StackOverflow post detailing that you need to specify the namespace for elements in the XPath query. Of course!

The following code is placed in to the project file that you wish to run the check in.

<Target Name="EnsureSQLScriptsAreEmbeddedResource" BeforeTargets="BeforeBuild">
    <XmlPeek 
        XmlInputPath="$(MSBuildProjectFile)" 
        Namespaces="&lt;Namespace Prefix='msb' Uri='http://schemas.microsoft.com/developer/msbuild/2003'/&gt;" 
        Query="/msb:Project/msb:ItemGroup/*[not(self::msb:EmbeddedResource)]/@Include">
            <Output TaskParameter="Result" ItemName="AllItems" />
    </XmlPeek>
	
    <ItemGroup>
        <Filtered Include="@(AllItems)" Exclude="SqlTemplate.sql" Condition="'%(Extension)' == '.sql'" />
    </ItemGroup>
    <Error 
        Code="SQL" 
        ContinueOnError="ErrorAndContinue"
        File="$(MSBuildProjectDirectory)\%(Filtered.Identity)" 
        Text="All scripts require a BuildAction of EmbeddedResource" 
        Condition="'@(Filtered)'!=''" />
</Target>

What’s going on here then?

  1. The XmlPeek task will take the project file and run the XPath query, collecting all matching attributes and sticking them in to a new parameter called AllItems.
  2. The XPath query works by looking at child elements of each ItemGroup element, matching against any that are not of type EmbeddedResource.
  3. For each of these, it will take the value of the Include attribute and use that to populate AllItems.
  4. The ItemGroup that follows will filter the AllItems collection to include only things which end with ‘.sql’. These will be stored in a new parameter called Filtered.
  5. Finally, the Error target will be invoked if any items made it through. Including the ContinueOnError attribute makes it possible to see all of the files that cause the build to fail, rather than stopping on the first one.

Props to StackOverflow users jessehouwing and Teun D for getting me on the right path!