Wednesday, April 9, 2014

Easy Automated Builds - Part 3

In Part 1, we discussed installing the custom activity DLLs into TFS so the build controller and agents can use them. In Part 2, we reviewed where the default build workflows reside in TFS and where we want to inject or Dynamics AX specific activities. So now, we're ready to actually build a workflow.

Before we begin, there are a few concepts that need explaining. When setting up a workflow, you can specify parameters to make it more generic. This will allow you to share the workflow across multiple build definitions, but run it with slightly different options. These are called "Arguments". Inside the workflow, you may need to store data in variables to pass from one activity to another in the workflow. For this purpose there are also "Variables".

In Part 2 we discussed the location to put the activities in. If you deleted the existing items as shown, you'll need to add a new "Sequence" activity. There are some basic "control flow" activities available in the TFS workflow: Sequence (a list of sequential steps), Parallel (a list of parallel tasks) and some others like foreach loops etc. These are listed in the Toolbox under "Control Flow". Open the ToolBox from View > Toolbox.



One of the differences again between TFS 2012 and TFS 2013 is that the build in TFS 2012 stores all the build information (build number, source directory, binary output directory, etc.) in variables in the workflow. In TFS 2013 everything gets stored in environment variables. The main idea with TFS 2013 is to be able to attach a PowerShell script pre- and post build (these are arguments to the workflow) so that you can do things without having to modify the workflow. So storing all the information in environment variables solves the issue of having to pass all that info to the PowerShell script.
What this means is that we need to add a few extra activities in TFS 2013 (skip this step in TFS 2012) to get some important information. The main things we want to get are the sources directory, the binaries directory, and the build number.


So for a TFS 2013 flow we need to add three new variables. You'll find the Variables list by click on the "Variables" tab at the bottom of your workflow screen. We will add three Strings and you can leave the scope to "Sequence" which should show up as your default.



Next, find "Team Foundation Build Activities" in the Toolbox, and drag & drop the GetEnvironmentVariable onto the workflow inside the new Sequence. When you drop the activity, you will be asked for the type you are retrieving. We will want a String.



For the purpose of this workflow, we want three of those, so drag three GetEnvironmentVariable activities, all three are strings. For each one, right-click the activity and select "Properties". In the "Name" property, we need to tell it which environment variable to get. You can find a full list here but the ones we need are:

Microsoft.TeamFoundation.Build.Activities.Extensions.WellKnownEnvironmentVariables.BinariesDirectory
Microsoft.TeamFoundation.Build.Activities.Extensions.WellKnownEnvironmentVariables.SourcesDirectory
Microsoft.TeamFoundation.Build.Activities.Extensions.WellKnownEnvironmentVariables.BuildNumber

You can give it a nice name for a DisplayName, something like "Get Binaries Directory". In the "Result" property, we give it the name of the variable we created, to store the string in. If you follow my screenshot, you'll have a BinariesDirectory, a SourcesDirectory and BuildNumber variable to assign to.


So with that done, we're back onto our actual work, including TFS 2012 and TFS 2013. Of course there are different ways to conduct the build, and I've outlined what we are using in this post but let's start with something simple here. First, I'm a big fan of "cleaning" an environment. Similar to a build from Visual Studio, you want to remove any artifacts from the last build, especially if you are using one AOS to do multiple builds for multiple code bases. But that brings up a point. You have to evaluate what it is you are trying to do exactly. For this example build I'd like to go "traditional" and assume that we are putting code in, and getting a binary (=ax model) out. So, no need to keep IDs or anything of the sort, we can just blow away all the code, and import all the code anew. To achieve this, you could just uninstall models from all the custom layers. That's what we initially did but for efficiency we changed to just restoring a base database and modelstore db. This has the advantage that you won't run into synchronization issues because you uninstalled models. It's also a bit more predictable in general, and a db restore doesn't take long at all (of course, or data-database can be empty - we don't care about data in this environment).

Before we get started, we need to get our custom build activities in the Toolbox. Open the toolbox and right-click somewhere on the window, and select "Choose Items".



On the "Choose Toolbox Items" dialog, open the "System.Activities Components" tab and click the "Browse" button.



Browse to the location where you saved the DLLs for Visual Studio's use, as discussed in Part 1. By default, this would be in C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PublicAssemblies (or 11.0 for VS 2012 and 10.0 for VS 2010). Once opened, you'll notice the CodeCrib.AX.TFS namespace activities showing up, and they should be checked by default. Click OK to confirm. Now you should see the activities show up under the "General" section.



So, for our workflow we first need to make sure the AOS is stopped. The "StopAOSServer" activity will either stop the AOS if it's running, or just not do anything if it's already stopped. Drag & drop the StopAOSServer activity into your workflow. Right-click it and select "Properties".

Brief intermission. All of the activities in the TFS library are meant for ease of use. That means they contain as many defaults as possible, but as many options as possibly needed in case you want to do advanced things. There are a few key things to understand:

1. Anything in these activities will be executed as the build agent user. So, the build agent user needs to be an administrator on the box to have access to start/stop services, it needs access to AX (as a user) for importing XPOs etc, and it needs direct SQL permissions to be able to modify the model store. If you do any sort of file system work like copying files, creating folders, etc - make sure the build agent will understand the paths and have access to them.
2. None of the activities REQUIRE any settings to say which AOS or which client configuration to use. By default, it will look for the current user (ie the build agent user) and that user's default configuration. From that configuration it will also find which AOS we're working with. However, all activities accept an optional "ConfigurationFile" property, where you can specify a config file. Same thing, it will read the config file to figure out which AOS account it needs to manipulate. If you want the fine details on all the activities' properties, check the reference page.

So, what I would urge you to do is to create an "Argument" where you can optionally specify a configuration file. Then, make sure all activities point to that argument and your workflow will be most flexible. So for this AOS STOP activity, I created a string argument "DynamicsAXConfigFile", and then here's what I have in the AOS properties:



You can optionally specify a time-out on stopping the AOS. Note that that time out is specified in minutes.

So, after the AOS is stopped, there is probably a bunch of things you can do in parallel. First, there's the matter of combining all XPOs from source control into 1 big XPO file (purpose is to have only one command to import the XPO, and also use AX's built-in logic to resolve dependencies between objects while importing). You can also call the "Clean" and "DeployReferences" activities, and then some other activity to either restore the database, uninstall the models, or do something of that sorts. Note that these activities may require the use of files (like the combine XPO activity). This is why we have the source directory and binaries directory stored in variables. A few more pointers to explain here:

1. The "Sources Directory" is the root folder where the build will retrieve all files. As we will see in our build definition setup in the next part, you can map different source tree folders onto different folders within the sources directory. For our builds, we've assumed that we never store anything directly in the root, but a sub-folder for anything needed for the build. For example, the XPO files for the model will be downloaded inside a "\Model" folder. Any third-party DLLs we need for compiling will be stored in "\References", etc.
2. Any files that you put in the "Binaries Folder" will be copied into the "Drop Folder" for the build. Basically, that is the output of your build. We surely want the axmodel exported to go there, but it can be handy to store the full XPO there as well. Or, perhaps you can make a boolean argument to indicate whether or not you want the XPO in the drop folder.
3. All property values are VB expressions, and the expected expression output type should equal the type of the property you are trying to set. So for the XPO file for example, you could set the "Folder" property to SourcesDirectory + "\Model".
4. Build number. If you want to provide the Build Number to the CreateModel activity - make sure to change your build number format to a.b.c.d in the build definition. We'll get to that in the next part.
5. For TFS 2012, the binaries and sources folders are already stored in variable names "SourcesDirectory" and "BinariesDirectory". Other details of the build are stored in a class in variable BuildDetail. You can use BuildDetail.BuildNumber. See the IBuildDetail info on MSDN.

So, here are the properties for my CombineXPOs activity:



Again, for an explanation of all the properties, their defaults and their usage, check the reference page.

So, here is where I leave you to play with the activities in the workflow. I didn't walk through a full workflow, however I prepared several built-out examples for different usage scenarios, from simple to very advanced. I hope you enjoy these examples, I'm planning to add a few more as well as improve the pages so you can see the properties on all the activities as well (as soon as I brush up on my JavaScript :-)

In the next part, we'll dive into creating build definitions and some concepts around that.

33 comments:

  1. I've made some good progress with your build libraries and nearly have everything up and running. However I am having an issue with the import XPO command, specifically the layerCodes parameter. I am currently passing this:

    New StringList("Var, g8XbNdLUbbDrciyhzlTFIA==")

    to the property but the build fails with the following error:

    Exception Message: Layer 'Var' requires an access code which couldn't be found in the Layer Codes argument (type Exception)
    Exception Stack Trace: at CodeCrib.AX.TFS.Helper.ExtractClientLayerModelInfo(String configurationFile, StringList layerCodes, String modelManifest, String& modelName, String& publisher, String& layer, String& layerCode)

    How should this parameter be formatted?

    Tthanks

    ReplyDelete
    Replies
    1. Good point, I need to document this somewhere other than in the code itself :-)

      Supports:
      var:CODE
      var : CODE
      varCODE
      var CODE

      Delete
    2. Cheers - got it sorted now.

      I have one further question though - how do you tell if a build has built cleanly without any errors?

      In the sample advanced workflow you have shared I can't see how you deal with this. It would be great if you could modify the AXBuildCompile activity to allow me to specify where we are logging to. The the log can be in the same folder as the output model.

      Having a way to check the build quality would be great as I'm working with multiple development AOSs and the build is working as a check that we are delivering working code.

      Thanks.

      Delete
    3. The build activities throw errors. The default workflow we're working inside of will actually fail the build on that. The compile errors, warnings and info are shown on the build information. It works exactly like a C# build works with TFS (ie, as you'd expect :-))

      Delete
    4. ah yeah - it works brilliantly - my builds just weren't containing any errors. doh.

      One thing I am struggling with is getting the work items to automatically associate with the build.

      In the log I get:

      "Associate the changesets that occurred since the last good build

      InputsUpdateWorkItems: True
      PreviousBuild:
      Enabled: True

      Warning: Cannot find the last label '': no changesets will be associated with the build.

      No changesets are submitted to build '1.3.0.19'."

      I can't figure out why the build process isn't able to find a good previous build. Do you know how to make this / what it is looking for?

      I found the below post but it hasn't really helped me.
      http://blogs.msdn.com/b/andy-lewis/archive/2011/01/31/how-good-was-that-build.aspx

      Thanks,

      Delete
    5. When we used TFS 2010 we didn't have any issues, we've had some strange happenings with our TFS 2012 install where sometimes items are not associated correctly with the build. I'm still unclear whether we are doing something wrong or if it's a TFS issue, but we have used a custom build report for a long time so it's never been a pressing issue to fix. And it seems sporadic for us.

      Delete
    6. I have figured it out. Possibly.

      I created a new build using the TFS 2013 template. This requires you to add a project path on the process tab of the build definition. Running a build using this template (not doing anything AX related) created a "good" build and associated work items next time round.

      I'm not sure if it's the project or having the "Run MSbuild" and "Run VS Test Runner" steps in the build process that fixes the issue - I am using both currently. I suspect it's just the project...

      Regards

      Delete
    7. hi (again). I'm having issues with the importXpo command. As the step is finishing it tries to parse the import log but the log hasn't finished being written to.

      Is there a way to add a timeout / wait to check to see if the file is accessible before we try to parse it? You can see in the temp dir that the file is only partially written. If I run the autorun file by hand it imports the xpo correctly with the correct logging.

      This is the stack trace:

      Exception Message: The process cannot access the file 'C:\Users\\Microsoft\Dynamics Ax\Log\ImportLog-c2e97594-5201-4574-9dc9-672981dcb850.xml' because it is being used by another process. (type IOException)
      Exception Stack Trace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
      at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
      at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
      at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
      at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
      at System.Xml.XmlTextReaderImpl.OpenUrlDelegate(Object xmlResolver)
      at System.Threading.CompressedStack.runTryCode(Object userData)
      at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
      at System.Threading.CompressedStack.Run(CompressedStack compressedStack, ContextCallback callback, Object state)
      at System.Xml.XmlTextReaderImpl.OpenUrl()
      at System.Xml.XmlTextReaderImpl.Read()
      at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
      at System.Xml.XmlDocument.Load(XmlReader reader)
      at System.Xml.XmlDocument.Load(String filename)
      at CodeCrib.AX.Client.AutoRun.AxaptaAutoRun.ParseLog()
      at CodeCrib.AX.TFS.AutoRunLogOutput.Output(CodeActivityContext context, AxaptaAutoRun autoRun, Boolean outputAllAsInfo, Boolean skipInfoLog)
      at CodeCrib.AX.TFS.ImportXPO.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)
      at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)

      Delete
    8. This was a red herring. My issue was the code in the database resore activity. Specifically "RESTORE DATABASE @DbName FROM DISK = @FileName WITH FILE = 1" my back up set was FILE = 3 so I was restoring an older db that had broken code in that caused the import to fail.

      All sorted now - thanks for supplying the source.

      Delete
  2. Hey Joris,

    I hope you are doing well. My name is Samuel and I’m the Managing Editor of Dynamics101.com, a recently launched training site for Microsoft Dynamics products.

    I just wanted to let you know that after careful consideration we have decided to include your site in our first annual list of Top 25 Dynamics AX blogs.

    I’ve mentioned your blog here http://www.dynamics101.com/2014/06/top-25-dynamics-ax-sites/, and we’ve created a badge that you could showcase on your site if you like. Just copy and paste the code from the bottom of my page.

    Either way, thanks for creating such a great blog! Keep up the good work.

    Best,
    Samuel Harper | Managing Editor
    www.dynamics101.com | @Dynamics_101

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. There was an update (v0.9.0.2) to the build DLLs that fix an issue with reporting compiler output for the client-side compile step... no issues with axbuild compiler output.

    http://dynamicsaxadmin.codeplex.com/workitem/1071

    ReplyDelete
    Replies
    1. Thanks for your posts. I'm currently working with TFS 2010 and I'm having problems loading the assemblies your published. Getting the error "Could not load assembly. Microsoft.TeamFoundation.build.client, version=11.0.0.0. Do I need any build extensions from 2012 or 2013?

      Delete
  5. Hi,

    Thank you for such a great set of tools. We're attempting to automate our ax2012 R3 builds with TFS2012. However, it keeps timing out on the importLabels step. We tried removing that step as a test, but then it just timed out on the next step. Occasionally we will get an info log on the client that says it cannot write AxUtil to the event log. Thoughts?

    ReplyDelete
    Replies
    1. this much time, I am assuming you found a solution?

      Delete
    2. We've since moved on to tfs2015. The issue corrected itself. I remember it being important to try and run the commands manually through the client. See if you have any popup windows that could possibly hold up the process.

      Delete
  6. Can you tell me how u add config file to aurgument

    ReplyDelete
    Replies
    1. That's a pretty broad question. All the activities that support it have a property to give the config file path. You could make a global variable for your build and pass that into all the activities. If you're asking where the "file" should be, that depends on your build machines. If you only have one build machine, you could put the file on its disk somewhere and reference that path. If you have multiples, you could put the file inside TFS repository so that it is download as part of the build (in the source folder), or you can put it on a shared drive and reference that path (assuming all of your VMs are on premise).
      Hope that helps. Your question isn't very detailed.

      Delete
  7. Okay, either I am going crazy or I am missing the smallest thing. I am running into this error in my build. Please look at the stack trace and let me know if you have ever seen this.

    Exception Message: Build does not support remote servers, client config server name () differs from current server (BUILDDEMO) (type RemoteServerException)
    Exception Stack Trace: at CodeCrib.AX.TFS.Helper.GetServerNumber(String clientConfigFile)
    at CodeCrib.AX.TFS.StopAOSService.Execute(CodeActivityContext context)
    at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
    at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)

    For some unkmown reason, it will not pull the server name from the config file. My setup is for a demo purpose. I am using VS Online for source and I have a build machine setup in my local on HyperV. I can get latest etc...all normal activities other than executing build

    ReplyDelete
    Replies
    1. Yeah there is a known bug (which is actually fixed in the other branch but I haven't released yet). Can you make sure the client config file it is using actually has a server name, instance name, port and all that set on the connection tab? There's some issues when some of the fields are empty it doesn't parse correctly and can't find the server name.

      Delete
    2. Any plans on releasing the update?

      Delete
  8. Hi Joris,

    I've happily been using your build activities for 2 years without any issue but now I'm running into performance/scaling issues.

    We have done all our dev in one model which is kept under version control. This is a problem as the xpo used in the build process is huge. It's taking about 2 hours to import (not ideal).

    Have you got any advice on this? Ideally splitting our code into multiple models and only building one smaller model would work best, but then we would still need to build one or more of them if we modify an existing object and have to keep track of the dependencies?

    I am a little bit stuck for ideas on how to approach this issue. Any pointers would be greatly appreciated. Thanks.

    ReplyDelete
  9. You'll want to use the official combine XPO tool instead of mine. The import re-imports objects in multiple passes to try and resolve dependency errors. The standard combine tool orders objects inside the XPO already in as best order possible to avoid dependency errors. That should speed up the import...

    ReplyDelete
  10. Hi Joris,

    We are in a similar situation. How do you tell the workflow to use the official combine xpo tool?

    ReplyDelete
  11. Hi Joris,

    When we use the ImportXPO process in our TFS 2015 Workflow, we are getting the following exception message. I thought that it might be due to an issue with XPO generated from the custom CombineXPO class but it's not. I get this error when I use the XPO generated from either the MS or Custom CombineXPO program. Have you seen this before? When I use axutil to just view the model contents, it looks like the object counts within the AOT are correct.


    Exception Message: Name cannot begin with the '>' character, hexadecimal value 0x3E. Line 1380, position 259. (type XmlException)
    Exception Stack Trace: at System.Xml.XmlTextReaderImpl.Throw(String res, String[] args)
    at System.Xml.XmlTextReaderImpl.ParseQName(Boolean isQName, Int32 startOffset, Int32& colonPos)
    at System.Xml.XmlTextReaderImpl.ParseElement()
    at System.Xml.XmlTextReaderImpl.ParseElementContent()
    at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
    at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
    at System.Xml.XmlDocument.Load(XmlReader reader)
    at System.Xml.XmlDocument.Load(String filename)
    at CodeCrib.AX.Client.AutoRun.AxaptaAutoRun.ParseLog()
    at CodeCrib.AX.TFS.AutoRunLogOutput.Output(CodeActivityContext context, AxaptaAutoRun autoRun, Boolean outputAllAsInfo, Boolean skipInfoLog)
    at CodeCrib.AX.TFS.ImportXPO.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)
    at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)

    ReplyDelete
  12. I've seen this before. When the import xpo code runs it will create an import log file that contains an error messages that contains the <> characters. When the build activity tries to parse the log it falls over.

    ReplyDelete
  13. Thanks for responding Nick62. How did you overcome this?

    ReplyDelete
  14. Find the log and see what it's complaining about and fix that and then do the build again. My autorun files get created in C:\Users\[BuildAccount]\AppData\Local\Temp these will have the path the output gets logged to. Mine is C:\Users\[BuildAccount]\Microsoft\Dynamics Ax\Log\

    ReplyDelete
  15. Thanks Nick, that's what I needed.

    ReplyDelete
  16. in my scenario i have Test, QA, Staging environments, so uninstalling models and adding them back would cause ID conflicts. Would it be best to restore the build machines model and data db first in the workflow with the target environment, run the build activities, and export a model store?

    I'm also a little confused on what to do with models supplied by ISV's. I am going to assume i create a folder and add them individually in the right order using your install model activity?

    ReplyDelete
    Replies
    1. I'm not sure why you would uninstall a model? You can build a new model, and then use that to replace the model on the existing environment's model store. Even if you want to outcome to be a model store (so you don't have to manually compile again after replacing the existing model) you'd need to build a model in a separate database first anyway - you can't import the XPO into the existing model store since it wouldn't DELETE or CHANGE existing objects. You have to be able to replace.

      As for ISVs - yes either install the models during build, or if you need to do something special during install put them in your baseline modelstore you restore prior to your build.

      Delete
  17. Do you have to do anything special for bilingual label files?

    I can create/deploy EN-US labels with no issues however if i use say fr-ca, errors are returned from the import labels activity on flush

    I can check-in the files with no issues and they are visible in the build directory after the import. It just seems AX doesn't like the file.

    The errors:
    Unable to load the specified language (fr-ca)
    Flush label WDP language fr-ca: false

    currently to workaround this issue i copy the files from TFS and install manually on the AOS every time i do a release.

    ReplyDelete
    Replies
    1. Hi Anthony. Sames was happening to me recently, and probably you already found out the problem. But just in case this is what I found. When the system creates the XML to flush the labels it checks every ALD file present on the sources folder, in my case axXXXfr-ca.ald is there, but for some reason it doesn't gets added. I debugged the build process and I found that if in the first 50 lines of your AXD file there is nothing specified, then it assumes it is empty.in my case I have 500 labels and only some in french around the 400 ID. So, in order to solve my problem I just added the french description for the @XXX1 label and that's it. It took me some time to figure it out :( But I was really exhausted to deal with label installation every time across 6 non-Prod environments. Hope this helps. I know, almost 3 months after but just saw it here :) Cheers :)

      Delete