Wednesday, May 9, 2012

Announcing: AX TFS Build Library

As you know I started a project on CodePlex last year, sharing our internally built scripts to automate builds for AX 2009. Obviously with the release of AX 2012 and new ways to release AX code (models!), we had to change our build scripts.

And with change comes the opportunity for improvement. We have taken the build process a step further and created workflow code activities for use with the TFS 2010 build server. If you want to dive into it, check out the beta release on CodePlex. Note this is a beta release, and particularly the AX 2009 code is not tested thoroughly.

What does this mean, code activities? Once loaded (see below), you can drag-and-drop Dynamics AX build activities in your workflow, and compose your own build template. Below screenshots from the AX 2012 build script included in the beta release.




This release includes the following code activities you can include in your build scripts:



There is a lot to talk about on this topic, including more documentation on how to use these activities. For now, feel free to download and try out the sample build template included! To be able to use these activities, you need to add the assembly DLL file to your source control on TFS somewhere, and point your TFS build controller to that server path, as explained in this blog post under "Deploying the Activity".


Stay tuned for more documentation, information, tips and walkthroughs on automating your AX builds!

36 comments:

  1. Hi,

    I have a question about the step: Clean Layer.
    What will happen with the object ID's in AX 2012 when you start each time from a clean layer to build your models from TFS...?

    As explained in the following article the object ID's can change.

    "Maintaining Installation-Specific Element IDs and Element Handles"
    http://technet.microsoft.com/en-us/library/hh352326.aspx

    Could you please explain how you handle the object IDs during the implementation traject of your project...?

    Kind regards,
    Mehrdad

    ReplyDelete
  2. Hi Mehrdad,

    I will make a point of explaining this further in upcoming posts. You need to consider this build environment a throw-away environment. The idea is that you build your code on a clean environment, so you have no artifacts that can influence your compile or code import. At the end when this is done, you have a model file as output, which has been checked and verified and is ready to move. You handle that model to move your code, not the build itself.

    So the point is, do NOT build in an environment that is used. Just like building a DLL, you do it on a machine on the side. When you're done, the model that is produced is ready to be moved. Then you can apply proper procedures, such as overwriting the existing model in an environment, so you don't lose ids. Again, just like a DLL. Build it on the side, deploy to where you need.
    If you use the build as your mechanism to move code, you're on the wrong path. The build is to verify (compile) and "package" what you are about to move.

    ReplyDelete
    Replies
    1. Hi Joris,

      Thanks a lot for your reply.
      So, as i understand when we export the built model and try to import it into an environment which has already an old version of the same model, without deleting it, the ids will stay unchanged.

      This is great. So, we are able to build each time from scratch.

      I want to use the opportunity to thank you again for the effort that you do for the improvement of whole the community.

      Grtz,
      Mehrdad

      Delete
  3. Hi,

    I am new to XAML and builds, but i already have a working, but fare not perfect automated build for our AX2012 Dev system in which most of the steps are done in powershell. I wanted to improve it by the help of your post, but could not make it work. I uploaded the dll and the XAML and put the folder on the build controler's "version control path to custom assemblies" field.
    When i am opening the XAML, I can see errors like this:
    "
    Error 1 Could not find type 'AXTFSBuildLibrary.CombineXPOs' in assembly 'AXTFSBuildLibrary'.
    d:\TFSRepository\Build\BuildProcessTemplates\CustomActivities\AX2012TFSLibraryTemplate.xaml 305 44"
    If and red rectangles showing "Activity could not be loaded because of errors in the XAML".
    I cannot see the AXTFSBuildLibrary in the toolbox.
    What could be the reason?

    Thanks in advance for your help,
    Kind regards,
    Peter Prokopecz
    peter.prokopecz@yahoo.co.uk

    ReplyDelete
  4. Hi Peter,

    the issue with the XAML is that Visual Studio doesn't have the DLL loaded so it can't display the activities in either toolbox or the XAML render.
    Having made the XAML inside the project of the build library, the DLL is referenced and loaded and you can edit the XAML within the scope of the project.
    Obviously it can't be the idea that you need the source to edit the XAML. All I could find on the subject is this blog post: http://blog.afsharm.com/2010/10/custom-workflow-activitys-problem-with.html

    ReplyDelete
  5. Hi Joris,

    Thank you for your work, it's been really nice to have it as a starting point. I have, however, run into an issue and I'm wondering if you have any ideas on it.
    What rights should the build service user have? Right now the CallClient function launches Ax32.exe but the activities take 0 seconds and not get anything done (but they finish without an error), while in a different domain I have seen those activities work fine.

    I am assuming this might have to do with service users ability to launch interactive processes, because if I run the parameter set the build actvities generate in a command prompt everything is fine and dandy.

    We are right now trying to get the local domain admins to allow our service users to have "Log on locally" rights, and I wonder if that will be sufficient...

    Do you think you could help us?

    ReplyDelete
    Replies
    1. We solved the problem in the meantime, and in case anyone else runs into the same: the build service user needs Log On Locally right, and it should log in at least once to start AX, if you installed the default installation, because the "Do you want to participate in customer experience program" screen blocks the execution of all the ClientExecute build activities, so they fail with timeout.

      Delete
    2. Sorry I didn't get back to you earlier. Yes, the experience program popup is an issue. Interestingly, the compile doesn't like it, but the synchronize works fine (or was it the other way around?)
      I'm not sure if log on locally would be enough since the service needs to be able to start and stop the AOS service.

      Delete
    3. As it is a test machine in a test domain, we werent really fine-tuning the rights, it got all kinds of admin rights that a build-service user should never, ever have, but the one we could not give to it (because it was controlled by group policy) was log on locally. So yes, it did have local admin as well :) It also needed rights to the SQL server, and ofc rights to the AX itself.

      For us, the build hung at the first clientexecute step, which was xpo import.

      Delete
    4. RE: customer experience popup: set SysUserInfo.SqmEnabled - see http://dev.goshoom.net/en/2015/01/improvement-program-dialog/

      Delete
  6. Joris,

    Can you tell me if or how you are setting the build version (Help > About)? I tried modifying the powershell build scripts for AX 2009 but I didn't have a lot of luck because I believe the AX32.exe process would sometimes remain open.

    Of course in AX 2012 the models now contain the version number in their manifest, so the code below may be slightly different for 2012, but it is still an interesting question.

    # "Setting build number..."
    # $axProcess = Start-Process -PassThru ($axClientPath + "`\Ax32.exe") -WindowStyle minimized -ArgumentList ($params + " -StartupCmd=setbuildno_" + $BUILDNUMBER)
    # if ($axProcess.WaitForExit($AXCOMPILETIMEOUT) -eq $false)
    # {
    # Throw ("Error: Build number update did not complete within " + $AXCOMPILETIMEOUT / 60000 + " minutes")
    # }

    -Joe

    ReplyDelete
    Replies
    1. For AX 2009 we always had the build script add a macro to the XPO before importing it. The Macro we then used to identify the build number. There's a few way to accomplish it. You could modify the about screen itself (Forms\sysabout), or you can mess with the ApplicationVersion class. We always modified the application version class in the slxapplall method. This looks at overlayered slxappl() methods and fetches the output of the method of each layer (basically sort of getting the build number of each layer). But by default, this only goes upto the USR layer. We modified that method to fetch upto the CUP layer, then overlayered the slxappl method to output the macro.

      The workflow items that i published have this feature to automatically add a macro containing the build number. The original powershell scripts do not, but it's easy enough to do. Just take an XPO of a macro from the AOT. Make the combineXPOs script output that text, and replace the contents with the build number.

      Delete
  7. Hi Joris,

    We ran into a rather interesting problem: due to a developers mistake the synchronize step failed. I was wondering how you deal with that? Basicly on the console session an Infolog is displayed with the error, and the build only fails after the timeout period passes. Do you have any ideas on
    a) how to notice that the build "hang" and is waiting for someone to press OK on the InfoLog, so the build break is noticed earlier
    b) how to get the error so it can be displayed as a reason for build failure? My current track to deal with the second question is parsing error trace file among the server logs, but im totally stumped on the first one. Synchronize executes in the kernel, and if it encounters errors it simply throws up and infolog and stops executing, and I did not find any way to control this behavior.

    ReplyDelete
    Replies
    1. Hi Gergely,

      you bring up a good point, we currently don't have a way to find out what happened automatically. We basically go into the build environment and check it out. Out-of-the-box I don't see how you could get more information from AX. With a minor customization you could of course log the infolog and even keep track of any forms that opened during the build.
      The issue I see with customizing that piece is that you no longer have a build that compiled on a vanilla AX, you have some sort of possible dependency on your customization.

      I have seen several issues with synchronize... 1) the modelstore changed and the noinstallmode flag wasn't set, or 2) the build user has the timezone or customer improvement dialog box come up and I've had some interesting issues using a config file that used the -Development flag to get into the development workspace.

      Delete
    2. Hi Joris,

      Our build process is finally (semi) complete. Here are the things that we learned and where we modified your template (thanks once again for posting it, it was a tremendous help).

      -The synch issue is very interesting, we talked to other dev teams and no one has a good solution. I spent quite a bit of time trying to modify the Synchronize command, but its a bit of magic, and we could only work around it:
      Basicly Synchronize doesn't only update the database, but calls the LoadAll method of a couple of classes to generate starting data or something, including NumberSeqApplicationModule which we derived another class from (without touching NumberSeqApplicationModule itself).
      This still caused AX to go all wonky, if we did anything with the original class, without calling Compile Forward on NumberSeqApplicationModule.
      So I added a new build activity which calls Compile Forward on that one specific class, so synch can run (also made it extensible, in case more of these pop up later)

      -I also modified the compilation step, to only recompile a single AX project instead of the entire AOT, so the build process takes about 30 minutes to complete. This requires the devs to add every piece of work into a project, analogous to Visual Studio slns, which is a bit of extra work, but we have a rigorous daily build process and anything that speeds it up is a bonus.

      -We also added a parameter importing step (all of these are extra SysStartupCmd options that are in the aot of the build server), that loads the dat/def files from a folder into the build server, so the configurators can check in their work as well.

      -We also integrated someone's solution (cant find it right now) to use x++ unit tests in visual studio testing

      -Also instead of clean layer, we are using a database backup restore strategy. Clean layer felt a bit wonky to us, which might be just a gut feeling, but this way we are sure to know whats exactly in the AX. Also, dont you need a synch after a clean layer?

      Im thinking about writing a full blog post about improving your build template, with proper references to everyone's work ofc, to make other people's work easier. Would you allow such a thing?

      Delete
    3. Hi Gergely,

      you can expect some updates to the scripts soon. A few things:

      1. the sync issue, we've had some issues but they were always related to dialog boxes (customer improvement, timezone, etc), and a few issues due to clean layer. In 2009 the XPO import contained IDs, so cleaning the layer without synching was never a problem, but now in 2012 it is since the IDs can be different.
      2. the clean layer is definitely something I would encourage, as well as a full compile. The idea is that you test one particular model. Compiling a project doesn't mean much, as that project could break any code, including standard, and you would never know. I agree nightly/daily builds are there to catch those, but still your build output is never guaranteed completely unless you did a full compile. Also, if the layer isn't completely empty, your compile could be bogus since there may be code in the environment that helps the compile, but once you extract the model it breaks (if the code isn't in that model).
      3. we do builds completely separate, nobody can use that environment. For testing, the build output (model) can be imported somewhere to test. This is good practice - as it also checks whether the model will install properly, everything is in it, etc. So yes you can in fact use import parameters etc, it is supported, but I won't put it in the standard workflow though.
      4. I have blogged about X++ unit tests using visual studio, so you may have actually used my code :-)

      I'm ok if you want to blog about this, but if you want to release code or sample workflows, I'd prefer you put it on the same codeplex project... I can give you access to that. I just want to keep everything in one place, including derivative works.

      Delete
    4. I added an extra build step since: Delete the goddamn .uac files, they can get corrupted occasionally, and they introduce really arcane errors when they do.

      Another issue popped up, this time with the imports (this was related to the corrupted uac files as well), so I wrote a small bit to parse the .importlog file to at least determine if the import was successful, and if it wasnt then stop the build. You wrote a very nice activity for the compile result parsing, now we only need a way to catch synch errors, then we are home free, and we will actually be able to notice if something goes wrong during the build:)

      2) We are working with a one model-one project solution, though this requires every dev to put every piece of work into a project, so we are trying to figure out how to build the model exclusively. The trick to deal with changed base classes is that if a developer changes something in default AX behavior, he should include those AOT elements in the project as well. This is rather volatile, but as this is the teams first AX project, we are trying to figure out what works and what doesnt.

      At the beginning of the build the layer is completely empty ofc, because we are always falling back onto a "virgin" state with a database restore.

      3) I completely agree with you on the separation of environments, the build server is the build server, not a playground. However loading the basic parameters (like the company, VAT categories etc) is required for unit testing, and a build is not a valid build for us until the developer unit tests ran on it. The general goal is to notice as soon as possible that something went wrong, before we go through the process of reverting and installing the multiple test servers (ie if even basic, unit testable functionality breaks, lets not waste everyone's time on waiting for the test servers to install)

      When (if) my pieces get into a clean enough state to be shared with others I will definitely use the codeplex project, there is indeed not much point in fragmenting the know-how.

      Delete
  8. How do I configure TFS for AX 2012 source code? TFS & AX is installed and both are mapped; but unable to figure AOT folder for TFS cofiguration.

    ReplyDelete
    Replies
    1. Hi Abhishek,

      I'm not sure I understand your question "unable to figure AOT folder for TFS configuration"?

      Delete
  9. Hi Joris,

    Ran into yet another nice issue: the labels dont get exported properly for some reason. (Or they dont get importerted properly, i have no way of checking whats inside the model file)
    Basicly only the labels that were added since the last build get exported. For a while I suspected the .ali files inside the servers directories, so Im deleting those as well during the build process, which sometimes seems to work, sometimes not, and that is not exactly the deterministic result I was hoping for..
    Also the whole thing seems a bit random, sometimes the label files get reimported nicely on the servers we deploy to, sometimes not, and Im a bit stumped. Did you run into anything like this?

    ReplyDelete
    Replies
    1. Greg,

      the labels are another known issue, and we have a fix for that. You are almost correct, but don't just delete the ALI, also delete the ALD and ALC from the server bin.
      It seems the AOS is checking the importing labels against the label FILE instead of the labels in the model store, so it ends up only importing new labels...

      Delete
    2. Hi Joris,

      Im actually deleting modelname.* from the C:\Program Files\Microsoft Dynamics AX\60\Server\MicrosoftDynamicsAX\bin\Application\Appl\Standard\ directory (if this is the one we are both talking about), both before importing the label files into the build server and before importing the model file into the deployment server.
      Deleting everything solved the problem on the build server, however after deploying the model on other servers, the labels are non existent.
      Do you know maybe any way to peek into an axmodel file to see whats inside? Im kinda curious if the export went awry, or the import doesnt do what it is supposed to...

      Delete
  10. Hi Joris,

    Comming back to the logged issue about the fact that AX 2012 Visual Studio Projects are not support in Workflow Activities (see also: http://dynamicsaxbuild.codeplex.com/workitem/473)

    As i understand actually there is no solution at this moment and the only way to solve the issue is to use the out of the box TFS synchronization functionality from within the AX client..?

    Am i right..? Or there are workarounds for this issue..?

    If yes, the question is if there is any issue when we choose to couple the build environment to the TFS and start synchronizing with the TFS from AX client using one or other startup command?

    I can understand that as a conceptual stand point maybe you do not want to couple your build environment directly to the TFS...but are there any other issue about this kind of solution..?

    Kind regards,
    Mehrdad

    ReplyDelete
    Replies
    1. Hi Mehrdad,

      we have in fact fixes for all of these things and I will be pushing those to codeplex in the next week or so. Most of these are workarounds for what I consider bugs in AX 2012. We have reported a few of these to Microsoft but not all yet. I'm also not expecting these to be resolved quickly which is why we went ahead and made workarounds.

      There is no real issue with synchronizing a build environment with TFS using AX's tools. As you mention, conceptually i'm not a fan but it works.

      Delete
  11. Hi Joris,

    There is one question about the need to do the CIL compile in the build environment.

    Is this only for a cross check or it is really neccesary...?

    Because as I understood we export the AxModel file from the build environment and have to still import the AxModel file into our ID-reference (staging) environment and from there we can export the AxModelStore after:
    - having a full compile
    - inclusive CIL generation
    - and checking if synchronization of the data dictionary can be done without any problem.
    This AxModel file will then ensure the minimum down time in the target environment.

    If this story is correct...would it not be usefull to extend the build template to also be able to generate an AxModelStore file...?
    Or this is conceptually not a part of build process...?

    Kind regards,
    Mehrdad

    ReplyDelete
    Replies
    1. There are rare instances where the X++ compile may run successfully but the CIL fails. We are not capturing the CIL compile output in the build script yet, but that's a feature we have planned.

      As for the model store. It makes sense to build a model and export that. Depending on your setup I could see you could save yourself a bunch of time by doing a model store instead.
      1) there may be other models that need to go in the environment. Remember the sample build template removes everything from the layer you're building, regardless of model. However, there is a feature to import models in the library, so you can add that. In fact, we've had the need to put that in the workflow template we're using currently.
      2) the library could support modelstore export easily. I'll have to check but it may already be implemented (but definitely not tested). I will add that as a feature request on CodePlex if it's not in the library yet. It makes sense to support this.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. Hi Joris,

      Building the AxModel each time from scratch is a very good idea.
      I think that we should keep this.

      Anotherways we will have problems with renamed and/or deleted objects.

      But is also very usefull (time saving) when we support AxModelStore export by the build process.

      Having these two points will led us to either:
      having one extra ID-Ref environment which should be used to import the AxModel file into it and performing the required steps to achieving the AxModelStore file
      -- OR --
      exporting the AxModelStore from the ID-ref environment and importing it to the Build environment and importing the already imported AxModel file into it (of course we should do some extra steps to achieve the end AxModelStore as described here above)

      Or you see another options here to be able to support the AxModelStore as the output of the build process..?

      Kind regards,
      Mehrdad

      Delete
  12. I'm actively working to update the CodePlex project with all the fixes we've made for several things, including the Visual Studio projects import, label issues, etc. I'm making sure not to break anything and adding fixes piecemeal, but I would advise waiting until I present out an actual release. Should be done by early next week.

    ReplyDelete
  13. Hi Joris,

    We tried to use the build template in a testenvironment, to see how things went. We had some small issues (permissions and so on), and finally are in the process of actually booting up the build client (yay :-) ).

    However, the synchronize step seems to fail intermittently. After 62minutes, we receive a timeout.

    I tried to login under the same account as the build controller (service acc), and tried a sync, which worked. Do you have any idea what can be wrong?

    Error:
    AX client execution of SYNCHRONIZE timed out
    Executing data dictionary synchronize
    Executing AX client with parameters ""C:\Builds\17\Sources\\DAX2012_DEV_FASE3\cus\AX60_BUILD_CUS.axc" -MINIMIZE -LAZYCLASSLOADING -LAZYTABLELOADING -StartupCmd=Synchronize"

    ReplyDelete
  14. When you logged in with the build user, have you made sure there were no dialogs coming up? (customer experience opt-out, timezone, etc). We have had some issues with that.

    ReplyDelete
  15. Yep, it had a customer experience dialog.

    Now synchronizing goes smooth, but I get an autorun timeout.
    Tried to increase the maximum timeout to 500minutes, because I wasn't really sure what it was doing. However after changing this timeout, the autorun still times out.

    If I check the model contents though, it seems as if everything is there. The generated XPO is about 30mb, would this be to much?

    ReplyDelete
    Replies
    1. Ok, on the environment that timed out, have you tried to go in right after the failure and do the synchronize manually? See if anything comes up there as far as dialogs, errors, etc...

      Delete
    2. I did, but the synchronizing was succesful, no warnings/errors ...

      Delete
  16. I am trying to implement the AX 2012 beta Build workflow. I am wondering what is being placed as the required Solution File and other Process parameters to get a successful build?

    ReplyDelete
    Replies
    1. Required model list is entirely optional. Since it cleans out all the models in the layer you're building it, you may have to re-import any third-party static models etc. That's what that is for.
      The configuration file is the config file that will be used to access the environment. So make sure the config file has the correct layer and layer code. The model signing key is entirely optional. This will be used by the model export to strong-name sign your assembly.

      So really, the only required parameter is the AXConfigurationFile

      Delete