Wednesday, April 9, 2014

Easy Automated Builds - Part 3

In Part 1, we discussed installing the custom activity DLLs into TFS so the build controller and agents can use them. In Part 2, we reviewed where the default build workflows reside in TFS and where we want to inject or Dynamics AX specific activities. So now, we're ready to actually build a workflow.

Before we begin, there are a few concepts that need explaining. When setting up a workflow, you can specify parameters to make it more generic. This will allow you to share the workflow across multiple build definitions, but run it with slightly different options. These are called "Arguments". Inside the workflow, you may need to store data in variables to pass from one activity to another in the workflow. For this purpose there are also "Variables".

In Part 2 we discussed the location to put the activities in. If you deleted the existing items as shown, you'll need to add a new "Sequence" activity. There are some basic "control flow" activities available in the TFS workflow: Sequence (a list of sequential steps), Parallel (a list of parallel tasks) and some others like foreach loops etc. These are listed in the Toolbox under "Control Flow". Open the ToolBox from View > Toolbox.

One of the differences again between TFS 2012 and TFS 2013 is that the build in TFS 2012 stores all the build information (build number, source directory, binary output directory, etc.) in variables in the workflow. In TFS 2013 everything gets stored in environment variables. The main idea with TFS 2013 is to be able to attach a PowerShell script pre- and post build (these are arguments to the workflow) so that you can do things without having to modify the workflow. So storing all the information in environment variables solves the issue of having to pass all that info to the PowerShell script.
What this means is that we need to add a few extra activities in TFS 2013 (skip this step in TFS 2012) to get some important information. The main things we want to get are the sources directory, the binaries directory, and the build number.

So for a TFS 2013 flow we need to add three new variables. You'll find the Variables list by click on the "Variables" tab at the bottom of your workflow screen. We will add three Strings and you can leave the scope to "Sequence" which should show up as your default.

Next, find "Team Foundation Build Activities" in the Toolbox, and drag & drop the GetEnvironmentVariable onto the workflow inside the new Sequence. When you drop the activity, you will be asked for the type you are retrieving. We will want a String.

For the purpose of this workflow, we want three of those, so drag three GetEnvironmentVariable activities, all three are strings. For each one, right-click the activity and select "Properties". In the "Name" property, we need to tell it which environment variable to get. You can find a full list here but the ones we need are:


You can give it a nice name for a DisplayName, something like "Get Binaries Directory". In the "Result" property, we give it the name of the variable we created, to store the string in. If you follow my screenshot, you'll have a BinariesDirectory, a SourcesDirectory and BuildNumber variable to assign to.

So with that done, we're back onto our actual work, including TFS 2012 and TFS 2013. Of course there are different ways to conduct the build, and I've outlined what we are using in this post but let's start with something simple here. First, I'm a big fan of "cleaning" an environment. Similar to a build from Visual Studio, you want to remove any artifacts from the last build, especially if you are using one AOS to do multiple builds for multiple code bases. But that brings up a point. You have to evaluate what it is you are trying to do exactly. For this example build I'd like to go "traditional" and assume that we are putting code in, and getting a binary (=ax model) out. So, no need to keep IDs or anything of the sort, we can just blow away all the code, and import all the code anew. To achieve this, you could just uninstall models from all the custom layers. That's what we initially did but for efficiency we changed to just restoring a base database and modelstore db. This has the advantage that you won't run into synchronization issues because you uninstalled models. It's also a bit more predictable in general, and a db restore doesn't take long at all (of course, or data-database can be empty - we don't care about data in this environment).

Before we get started, we need to get our custom build activities in the Toolbox. Open the toolbox and right-click somewhere on the window, and select "Choose Items".

On the "Choose Toolbox Items" dialog, open the "System.Activities Components" tab and click the "Browse" button.

Browse to the location where you saved the DLLs for Visual Studio's use, as discussed in Part 1. By default, this would be in C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies (or 11.0 for VS 2012 and 10.0 for VS 2010). Once opened, you'll notice the CodeCrib.AX.TFS namespace activities showing up, and they should be checked by default. Click OK to confirm. Now you should see the activities show up under the "General" section.

So, for our workflow we first need to make sure the AOS is stopped. The "StopAOSServer" activity will either stop the AOS if it's running, or just not do anything if it's already stopped. Drag & drop the StopAOSServer activity into your workflow. Right-click it and select "Properties".

Brief intermission. All of the activities in the TFS library are meant for ease of use. That means they contain as many defaults as possible, but as many options as possibly needed in case you want to do advanced things. There are a few key things to understand:

1. Anything in these activities will be executed as the build agent user. So, the build agent user needs to be an administrator on the box to have access to start/stop services, it needs access to AX (as a user) for importing XPOs etc, and it needs direct SQL permissions to be able to modify the model store. If you do any sort of file system work like copying files, creating folders, etc - make sure the build agent will understand the paths and have access to them.
2. None of the activities REQUIRE any settings to say which AOS or which client configuration to use. By default, it will look for the current user (ie the build agent user) and that user's default configuration. From that configuration it will also find which AOS we're working with. However, all activities accept an optional "ConfigurationFile" property, where you can specify a config file. Same thing, it will read the config file to figure out which AOS account it needs to manipulate. If you want the fine details on all the activities' properties, check the reference page.

So, what I would urge you to do is to create an "Argument" where you can optionally specify a configuration file. Then, make sure all activities point to that argument and your workflow will be most flexible. So for this AOS STOP activity, I created a string argument "DynamicsAXConfigFile", and then here's what I have in the AOS properties:

You can optionally specify a time-out on stopping the AOS. Note that that time out is specified in minutes.

So, after the AOS is stopped, there is probably a bunch of things you can do in parallel. First, there's the matter of combining all XPOs from source control into 1 big XPO file (purpose is to have only one command to import the XPO, and also use AX's built-in logic to resolve dependencies between objects while importing). You can also call the "Clean" and "DeployReferences" activities, and then some other activity to either restore the database, uninstall the models, or do something of that sorts. Note that these activities may require the use of files (like the combine XPO activity). This is why we have the source directory and binaries directory stored in variables. A few more pointers to explain here:

1. The "Sources Directory" is the root folder where the build will retrieve all files. As we will see in our build definition setup in the next part, you can map different source tree folders onto different folders within the sources directory. For our builds, we've assumed that we never store anything directly in the root, but a sub-folder for anything needed for the build. For example, the XPO files for the model will be downloaded inside a "\Model" folder. Any third-party DLLs we need for compiling will be stored in "\References", etc.
2. Any files that you put in the "Binaries Folder" will be copied into the "Drop Folder" for the build. Basically, that is the output of your build. We surely want the axmodel exported to go there, but it can be handy to store the full XPO there as well. Or, perhaps you can make a boolean argument to indicate whether or not you want the XPO in the drop folder.
3. All property values are VB expressions, and the expected expression output type should equal the type of the property you are trying to set. So for the XPO file for example, you could set the "Folder" property to SourcesDirectory + "\Model".
4. Build number. If you want to provide the Build Number to the CreateModel activity - make sure to change your build number format to a.b.c.d in the build definition. We'll get to that in the next part.
5. For TFS 2012, the binaries and sources folders are already stored in variable names "SourcesDirectory" and "BinariesDirectory". Other details of the build are stored in a class in variable BuildDetail. You can use BuildDetail.BuildNumber. See the IBuildDetail info on MSDN.

So, here are the properties for my CombineXPOs activity:

Again, for an explanation of all the properties, their defaults and their usage, check the reference page.

So, here is where I leave you to play with the activities in the workflow. I didn't walk through a full workflow, however I prepared several built-out examples for different usage scenarios, from simple to very advanced. I hope you enjoy these examples, I'm planning to add a few more as well as improve the pages so you can see the properties on all the activities as well (as soon as I brush up on my JavaScript :-)

In the next part, we'll dive into creating build definitions and some concepts around that.

Tuesday, April 8, 2014

Easy Automated Builds - Part 2

In Part 1 we looked at installing the TFS custom activities into the source tree, telling the build controller where to find the DLLs. Finally we also made sure Visual Studio can find our DLL files so we can edit workflows.

One of the major updates between TFS 2012 and TFS 2013 is the default workflow. One, it is stored differently. And two, the 2013 workflow has been simplified to basically a one-page workflow (TFS 2012 has pages and pages of collapsible workflow steps). To account for both, let me start with showing you where to edit the workflow. Again, 2013 is super-easy. I like to start with the bad news first, so let's start with TFS 2012.

Also, before you continue make sure you have the custom activities DLLs available for Visual Studio, as explained in Part 1.

In TFS 2012, open your source control explorer and find the project you are building for. Each project will have a folder called "BuildProcessTemplates". You will find a few workflows in there, including a "DefaultTemplate" with some version number appended to it. Double-click the Default Template XAML to open it. Once it's open, from the File menu select SAVE AS and create your own name, such as AXBUILD.xaml or something more creative. Now, I've seen regular issues with Visual Studio giving errors on even the default workflow file. If you see any warning icons (red or yellow triangles with an exclamation point) or if you see any bright red error messages inside the workflow - just close and re-open the XAML file. I'm not sure if that is just my system, but regularly I have to open a workflow twice to make it load all perfectly.

Now that you've loaded the XAML file, you'll notice how large it is. To find the spot we need to change (ie, where Visual Studio projects are being compiled), click the "Collapse All" link at the top right.

Now, one by one, open the nodes I'll list out by clicking on the arrows.

Open the following nodes, one inside the other:
1. Run On Agent
2. Try Compile, Test, and Associate Changesets and Work Items
3. Sequence
4. Compile, Test, and Associate Changesets and Work Items
5. Try Compile and Test

You could go down deeper and keep the testing piece in. This would allow you to actually make a generic Visual Studio test project part of your build, which can call into AX unit testing, as explained here. For the sake of simplicity here, I'll stick to interjecting here (you'd need to go down a few more levels). At this point, you can delete the "Compile and Test" node entirely (leave the encompassing "Try Compile and Test"). This is the place where we will add our own Dynamics AX build workflow, inside the try compile and test.
Note that the way the workflow is set up prior to where we stepped in, you will still need a solution file (.sln) for the build to work. You can just create a blank solution (without projects!) and add that into the source tree along side your code. In fact, just opening the XAML file without a project in Visual Studio will create a blank solution because VS needs context. If you look at your solution explorer you'll notice VS has created a blank solution (probably "Solution1" or something) without any projects in it. You can just save that solution (give it a proper name) and put that in your source control tree. We'll need it later when we setup a build definition.

In TFS 2013, the story is different. For starters, each project no longer has XAML workflows copied into it when you create it. There are default templates still, but they are stored on the server somewhere and not copied down to the project level. So, to customize a default template, we first need to grab it. To do this, we first need to start a new build definition, and we can download the default template from there. You need to get to "Builds", and click the "New build definition" link. Technically you can work from any version of VS to connect to any version of TFS - however I was unable to grab the default template for TFS 2013 from any other version but VS 2013. If anyone knows how, please put it in the comments!

When you're on the new build definition screen, go to the "Process" tab and click the "Show details" arrow. In the details you can click "Download". Close the new build definition and don't save it, we'll come back to this later.

Now, open the XAML file you just downloaded. If you get any sort of error, close it and open it again. As I explained in the TFS 2012 section above, I've seen some interesting issues the first time you open a XAML file. Loading it a second time seems to load all dependencies correctly. Now, this workflow is a lot more condensed than the TFS 2012 one. What we are really looking for is the "Run on agent" > "Try" > "Compile, Test and Publish" section.

As with the TFS 2012 build, you can delete the whole Compile, Test and Publish structure and we can insert it there. If you want to include test projects (again, see TFS 2012 explanation above) you can drill down into this sequence and remove the msbuild compile steps with our Dynamics AX scripts, and leave the testing in.
So, now you have another blank slate to start building our Dynamics AX build workflow!

Ready for Part 3, build a workflow!

Monday, April 7, 2014

Easy Automated Builds - Part 1

I'm glad to announce updates to my Dynamics AX Admin Utilities, including a brand new build library. It's been in use here at Sikich for quite a while now, and with our latest modifications I'm glad to release the binaries (although as always the code is available at any time). I want to make sure to thank my partner in crime, Dan, for helping me out with the code and the testing it here internally. I also got some good feedback from a fellow MVP, Kenny Saelen, who's also implementing these activities and got me the final bits I needed to finally implement async. Thanks!
Now, these activities can be used with or without TFS. Yes, that's right. They are built against the TFS client libraries for optimal use with TFS. But I will show in an upcoming post that you can use these activities using regular .NET workflow as well.

The major difference with our previous activities is that these are 1) based on the work we've done around the admin utilities and 2) they are optimized for ease of use and minimal configuration of options. I will outline an example here using Team Foundation Server 2013 with AX 2012 R3. But note that we use the exact same libraries internally for AX 2012 RTM, AX 2012 FPK and AX 2012 R2 releases, using Team Foundation Server 2012. I'm fairly confident these activities will also work with TFS 2010 - however you may need to get the source and recompile the libraries against the TFS 2010 client DLLs. I will test this at some point. However, there is nothing special in the activities that is specific to TFS 2012 or 2013. I am pretty sure these activities will NOT work in TFS 2008 since a lot was changed in TFS going into 2010.

If you want to dive right in, download here. Reference documentation is here.

First, you'll need to download the binaries from CodePlex. The TFS release package includes all the needed DLLs (basically the dependencies on the admin utilities). There are no separate versions for AX RTM vs R2 vs R3, and no separate versions for TFS 2012 vs 2013 (the DLLs are built against TFS 2012 and work fine with the 2013 DLLs).
Once you have downloaded the zip file, you will need to put the DLLs in a project inside the TFS Project Collection where you will be doing builds. If you have multiple collections, you'll need to put the DLLs inside each one.

I chose to create a separate project called AXBuild and inside a folder called "Assemblies" I uploaded the activities. I doesn't matter where exactly you put them, as long as they are in the source tree somewhere.
Next, we need to go into the TFS Server Administration Console, open up the Build Configuration.

A few things to explain here. A build service is the process that accepts requests from clients to perform a build. A build controller orchestrates the builds and delegates the heavy lifting to a build agent. A build agent is the process that actually performs the build itself. An agent is registered with one controller only, the controller is tied to a service, the service is set on one specific project collection. You can run many services and controllers and agents, it's just important to note the roles they play. If you are just playing around and have only 1 thing to do with TFS and that's AX, you can stick with one collection, one service, one controller, and one agent :-)
Now, if you haven't installed a service/controller/agent yet, go ahead and run through the setup. If you have multiple agents (the default during install), you can run multiple builds at once. HOWEVER, keep in mind that each build will of course require its own AX AOS server! You don't want two builds running and trying to use the same instance of AX. Next, the agents should be installed on the AOS machine, so that the build process runs on the same server as the AOS. We may support remote builds at some point, but right now that's the way it goes - and it's really not an issue. In our setup we have a build service and controller running on the same machine as TFS itself. Then, we have specific build machines that just run one agent, connected to the main controller.

With that out of the way, the build controller is where we need to point to our custom activity assemblies. This of course means if you need these for multiple controllers, you'll have to set each one up to point to the assemblies. Also, make sure you log on to the machine with the build controller - in case you run the agent on a different machine. On the controller, click the Properties link. On the dialog that appears, you can set the path to your custom assemblies. You can look up the correct path by clicking the ellipsis button there.

Click OK and that is all we need to do here. We're now all setup and ready to create a build workflow. TFS has workflows that handle everything that is needed in a build. It creates a source label (if you don't know what that is, no problem), it grabs all the code, associates the work items from the changesets to the build, etc. And of course, it runs the compile. Now, the default workflows that come with TFS of course assume that you will be compiling a Visual Studio project. So, we want to take one of these pre-built workflows, remove the visual studio compile stuff, and add in whatever activities you wish to use to build AX.
Now, if you are using TFS 2013 (or Visual Studio Online), this is a WHOLE lot easier.

To edit the workflow, we will be opening a .NET workflow for TFS inside Visual Studio, and drag&drop our activities and set the properties. For this to work, Visual Studio needs access to the DLLs for the activities, of course. There are two ways to do this. You can just put the DLL files in the PublicAssemblies folder of Visual Studio ( in C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies - replace your Visual Studio version with 10.0 for VS2010, 11.0 for VS2012 and 12.0 for VS2013). The other option is to actually create a VS project (let's say a C# class library), add the DLLs as a reference, and open the workflow in the context of this project. I personally find it easier to copy the DLLs in the public assemblies folder, but that's my choice.

We are now ready to create a workflow, which will be Part 2.