Wednesday, June 29, 2011

AX & TFS FAQ

This is my first post containing an FAQ on the AX and TFS integration. I get a lot of the same questions, and below are the most commonly asked ones. This is a lengthy post so I apologize to the RSS subscribers.

Feel free to leave your questions in the comments, and I will keep this FAQ updated.



AX & TFS FAQ

Q: How to move one X++ project at a time, and what if it contains objects that have been modified for other X++ project as well?
A: This is where TFS' intelligent branching/merging process comes into play. The prerequisite is that you check in code for one X++ project at a time. That way, TFS knows which changes are "bundled" together. Obviously you can multiple changes over multiple days, but it is imperative that you never check in changes you made for different projects.
When you merge a change set to another branch, which contains objects also modified by other projects that you are not moving, TFS will raise a flag. It will allow you to make a decision (on the XPO text) whether you wish to move the whole thing anyway, whether you want to move just the change associated with the change set you're moving, or whether you wish to manually modify the change to make it work. This works well for code in general but I always manually check to make sure anyway.


Q: How to build one project or one object or one changeset?
A: Although I understand the question in certain situations, I feel like there is never a good enough reason not to build the whole thing (except time perhaps). In any case, this basically boils down to creating a custom build workflow. When building just one object or one changeset, I'm fairly certain that can be done with the workflow. To build just one project, that is more tricky since TFS is not "aware" of projects, for what it's concerned, all it has is a text file (xpo) dubbed the "project definition". That being said, consider what we're doing with builds today: we are taking all XPOs and combining them into one giant XPO to import. What can be done of course it instead of combining all XPOs, first read a project definition, then combine only the objects described in that. Then follow the normal process for the build.


Q: How can we build a patch/hotfix layer?
A: This relates to the previous question. If you have a project containing the objects touched for your patch, you can use the combine-xpo approach I described above. If you want to build certain changesets only, either build a custom workflow, or consider perhaps a different branch, which can then be completely built in a patch layer…


Q: How can we have multiple developers work on the same AOS using TFS?
A: Due to the way the standard AX integration is designed, this is not possible. Of course in reality, this is a VERY common scenario, and Microsoft has asked a lot of questions and asked for lots of feedback from me and other in the community to try and solve this issue in AX2012. The solution unfortunately, has not worked it's way into the product. Instead, we will get more access to the integration code with TFS (the .NET "proxy" assembly is now a visual studio project in the AOT) so we can make any changes we need to ourselves.

A: That being said, in our AX2009 environments we do have multiple developers on one AOS. This is done through some customizations to AX, and some NTFS (windows filesystem) tricks. Bottom line:
- We modify the AX integration to append the AX username to the local repository folder, this satisfies TFS that all users are using a different repository "working folder"
- We add NTFS "junctions" so that although each user has its own folder, they are physically all the same folder (think sym links in unix/linux). This satisfies some issues with AX (namely if you have outdated XPOs there's a danger of updating the AOT with old code when you undo a chance for example).


Q: Can AX2009 support branching in the source tree? IE can I point AX 2009 to use a SUB-folder in my source tree rather than the root?
A: Standard AX 2009 will always put all the files starting at the root. Please note that this has been fixed in AX 2012 and you can specify a branch folder there.

A: For AX 2009, we have branching working at our offices at Streamline Systems. The only thing you really need to do is, after AX sets up your workspace, is changing that workspace (using visual studio) so that the working folder on the server points to your sub-folder. Of course, it only takes one user to not do this properly to mess up your source tree. To that end, we have automated this "fix" on the workspace by adding some .NET code in AX that gets executed when you log into AX, and changes your server working folder.


Q: In ISV scenarios, how can we maintain multiple components, and create unique builds for clients containing a unique combination of our components?
A: To be honest, I do not have a good answer to this. For big components, it makes sense to develop them in separate AX environments. For small ones, that sounds like overkill. In any case you can hook them up to the same ID server to avoid object ID collisions.

A: As far as branching is concerned, this would require a branch for each unique combination where you merge in the components you need. That is technically not unrealistic, but may be too much work for your scenario. Especially if you then need to branch for each release off of that…

A: I believe the AX developer community needs to start thinking more in lines of general software development practices, which in the past has never been the case. There are issues due to features and architectural challenges that are unique to AX, but most of the time those can be overcome. In this case, think of each component as a DLL or software product you would sell to a client. The uniqueness in AX is that they will all go in the same layer. But honestly, that's an installation detail, and it should not prevent you from considering each component as a unique product in your development cycle.

A: With the coming of AX 2012, the introduction of models, AxModel deployments and installations, I believe it will become easier to do these sorts of things.


Q: In client scenarios, clients may be doing their own development in a higher layer, how does that work?
A: When clients do development of their own, it should go in a higher layer than your delivered solution. This makes sense from a maintenance/support and liability point of view. We usually deliver our builds in a VAR layer and recommend customers use the CUS layer for their customizations.

A: In situations where you do off-site development and move layers into client environments (are current setup at Streamline Systems), it is true clients may be over-layering your objects so that any changes you make are not reflected in the final code layer "combination". To that end, you have a build log from TFS that can tell you what objects have been modified since the last release, and you have a layer compare in AX to detect layer conflicts. This sounds like a lot of work, but it isn't really. The point of source control is that you know exactly what has changed, so unless you have a ton of changes every release, the layer merge effort should be minimal.

A: In situations where there is development in multiple layers at the same time (such as traditional client projects where all development is done on-site), there are unique challenges. Even though for example fields on a table can be in different layers (ie not the whole table is in a layer at the time, like reports or forms in AX 2009), from a source control point of view the whole objects is moved into the higher layer. This prevents you from making ANY changes to the object in a lower layer as soon as it's over-layered. I have had numerous discussions with Microsoft on this subject, but there is no good way of resolving this.


Q: Why do we want to move layers and not XPOs as we've always done?
A: My favorite topic. Moving XPOs is literally moving source code (the XPO is a plain text file containing the source code). There are several issues with that approach:
1) the person importing the XPO can make mistakes (merge issues, object IDs, "delete tables and class members", etc). An XPO also implies someone took an export somewhere, which is the same issue, mistakes could have been made in the exporting.
2) from an auditing perspective this "should" be unacceptable: whomever imports this into production can modify the code, which is a segregation of duties issue, and a liability since after import the code cannot be guaranteed to be exactly the same as the test environment (export/import could introduce changes, merge issues, etc)

A: Usual complaints are one needs to take down the AOS to move in a new layer. My response to that is taking down the AOS, moving in the layer and starting it up again is a matter of minutes. Proper procedure when moving XPOs would be that all users are out of the system anyway, and that you do a full compile afterwards to make sure there are no issues. The difference is minimal, and not an excuse to keep doing XPO exports.

A: Moving compiled, binary code is the approach anyone should take. You do not receive hotfixes for Windows from Microsoft in source code form, you receive binary updates (new DLLs, EXEs, etc). This is also where Microsoft is going in AX 2012. There are models, and model exports (.AxModel). XPOs in AX 2012 are not the recommended way of moving code.

A: By moving code in layers, they can be versioned easily as "releases", which makes it easy to keep track of what version of the code is deployed where. XPO moves (unless extremely strictly controlled) have the potential of having different versions of different objects. Layers are the only guaranteed way of having two environments use the same code.

20 comments:

  1. Great site and great posts. These have been most helpful.

    We've run into a problem in our AX 2009 and TFS 2010 integration where it seem to be assigning and creating workspaces on the fly causing syncronization errors and other problems.

    My question is, what is responsible for creating and assigning a workspace to a particular user and AX client? The TFS setup in AX doesn't specify the workspace and when I look in the TFS workspace table [Tfs_DefaultCollection].[dbo].[tbl_Workspace] I find that every developer has from several to dozens of workspaces, all in the form _AXWORKSPACE. When we originally set up our VMs, we had to open Visual Studio and assign the workspace that was already there to a local repository to match what was specified in AX.

    What is creating these other workspaces and why would it be creating multiples?

    Thanks so much for your help.

    Steve

    ReplyDelete
  2. Hi Steve,

    the only times I have seen these workspace re-created over and over again is when some type of error occurred when creating the workspace or with the initial setup in version control of the user. A local repository that didn't exist, perhaps the user doesn't have access to the TFS project and it can't download the latest version, that sort of thing.
    The creation of the workspace is actually handled in a proxy .NET assembly (DLL). This unfortunately means there's not much you can do about this behavior, except trying to find out what the error is so you can correct it.
    I believe there was a flat file somewhere in your local settings that contained the next number it will try for the workspace - in case you want to reset it. However, I'm currently unable to find where I read that!

    FYI, in AX 2012 this proxy assembly is now a AOT Visual Studio project, which means you DO have access to the code, with possibility to change it.

    ReplyDelete
  3. Thanks for the information. There are still some issues with our TFS/AX integration that seem a bit unusual but we're fairly certain that the extra workspaces were created under the following or similar circumstances:

    Each developer was given a new VM with AX installed and enabled but not pointing to the correct repository path.

    However, if they opened AX the first time, it would automatically create a workspace in TFS associated with their machine and the repository.

    Then, when they either created a new workspace in Visual Studio and mapped it to a local respository folder or if they went into AX to set the repository, it would create a second workspace in TFS for this machine.

    ReplyDelete
  4. Is it possible to have two (or more) AOS instances on the same tfsbuild server, each AOS mapped on its own TFS Branch in order to build them nighlty? Our challenge: 3 Dev teams using their own respective Dev branch, plus one Main branch and one Maintenance branch. Our goal: having each branch built (at least) nightly within their own respective AOS instance. Or maybe it's a really bad idea and we should have a tfsbuild server for each branch and AOS??? Any hint would be appreciated... Thanks! :)

    ReplyDelete
    Replies
    1. I'm not sure what you're asking. You're saying that you want to build within their own AOS instances but then you ask if maybe you should have an AOS per branch, which is what you said before. Also, define "tfsbuild server" in your scenario, as it could mean different things since it's not an official term.
      Either way, you can do whatever you want, all possible scenarios for your questions i don't see problem in doing?

      Delete
  5. Hi Joris...

    At first... Really great and helpful site. I'm trying to configure AX 2009 with TFS 2010 (Don't ask me why we still using 2009 please... that is a veeery long story :-) ). I'm trying to have "One AOS - multiple developers" environment. I've spotted that you solve the problem using "This is done through some customizations to AX, and some NTFS (windows filesystem) tricks.". Would share me more details about the solution. What did you change in AX? How to configure TFS properly? I'd be very glad for your help.

    Thanks
    Sebastian

    ReplyDelete
    Replies
    1. My email address is: smankowski@outlook.com

      Thanks

      Delete
    2. I'm pretty sure i have more detail on this somewhere on my blog but I can't find it right now. Anyway, we've customized AX so that it adds the AX userID to the end of the local repository name. So if your repository is C:\XYZ then it will add "Admin" to it for example, to make C:\XYZ\Admin . This is fairly easy to customize.
      The problem you get then is that the AOT will always have the latest code, but your local repository may not (if another developer changed it - you wouldn't have the XPOs). You can instill procedures to always do get latest etc but that can present challenges too. So what we did is when a user first logs on, AX will create the workspace. Since we add the UserID we have to create the folder for that user first, or TFS will complain that it doesn't exist. So what we do is instead of creating the folder, we create an NTFS junction of one main folder. So we have a folder called "Junction" for example, and each folder we create for a user is a junction off that. This means that each user will have its own folder, but it's technically the same on disk. That way, although TFS will still think you're outdated, you do actually have the latest XPO on disk which avoids some issues.

      Now, if you do that, you have another issue :-) Because AX looks at the file to see if something's checked out (read-only versus read/write) instead of looking at AX. So the AOT will allow you to edit something even though it's checked out to someone else. This can be customized as well, there's some "can edit" method which you can have check TFS instead of the file system (AX actually has this feature when you check for pending objects, it checks what's checked out to you).

      So, one customization opens a new problem that leads to another customization, etc. It's not pretty - and even with the customizations you can sometimes run into strange issues. So it's PARAMOUNT that you understand what you are doing. This is part of why I haven't posted the code anywhere: it's still not the perfect solution and people who would blindly take the code will probably run into trouble. On the other hand, if you understand exactly what I'm saying, you're probably OK implementing the changes yourself and consequently understand the limitatons...

      Delete
    3. Joris... your reply lead me on the right way :). I believe I've sorted most of the problem but I've stuck at one. It's not very important and we can live with it, but... it's little annoying :). TFS supports mutliplecheckouts. Everything is fine until the "second" person tries do the checkin. At first infolog error appears saying the file is write-protected or in use - the reason is that first checkin changes the attribute of the local file to "read-only". The second person is unable to overwrite local ".xpo" file. At next standard TFS "resolving-conflict" window appears. Just click OK... and infolog says that everything went fine. Unfortunately the "second" checkin is not visible in history. Did you sort it out somehow? I know that easiest way is to just lock multiplecheckouts by changing SysVersionControlFilebasedBackEndTfs.supportMultipleCheckout() method to return false :).

      Delete
    4. We changed the canEdit method (or something like that) which by default just checks for the read-only flag on the file, to explicitly check TFS when the read-only flag is NOT set. Then it checks TFS to make sure the file is actually checked out, and if it is but the user that checked it out is different the user making the new checkout request, it returns false.
      I can check on details but I'm pretty sure that's what we have.

      Delete
    5. That is exactly what I did. But TFS allows checking out object for more than one user. So if you asked TFS for the user the file is checked out to, it returns more than one user. That means more than one user can edit the file => i.e. class in AX. At this moment I modified commandCheckIn method to remove the read-only flag and add it again after checkin process is completed. I didn't test it yet... but I'm hoping it's gonna work. Let you know what discovered.

      Delete
    6. @AXArchitect, I had a similar issue and I think the reason the second check-in is not visible in history might be due to the physical file location in your setup from different users or different models. I believe that since these are flat files, the XPO can only exist in one place.

      In my scenario, I was in model "A" and somebody else in model "B", which were different folders in TFS, and then another developer turned off his TFS briefly or something odd. What resulted was the XPO file was created in both of our model folders, which I don't think should happen. I looked at the history of the object on 4 different developer's screens and 3 saw the same history and 1 saw a different history. Put breakpoints in \Classes\SysVersionControlSystemFileBased\getItemHistory and \Classes\SysVersionControlSystemFileBased\fileName and see if the filename/path returned for each developer is the same or different.

      Delete
  6. Hi,
    Do you have more details on the change you made in AOT to check out that forces a check on actual TFS instead of the file.

    ReplyDelete
    Replies
    1. I need to dig up the code to get details, but I know we looked at the screen "pending changes" that exists in AX, and re-used a lot of that code.

      Delete
  7. Let's say we have two developers (A & B). "A" creates 5 fields in Model A on the CustTable and checks-in to TFS. "B" is working on a large project in Model B and realizes after examining CustTable that those fields belong in his model. B clicks on the un-checked-out CustTable and selects the fields and does "Move to model B". It works on his screen, but no TFS messages pop up, and there are no pending changes.

    Developer "A" sees the fields still in Model "A" after a get-latest, db sync, and compile of CustTable. Is there any way to easily resolve this?

    ReplyDelete
    Replies
    1. Different models into TFS from the same AX instance just plain doesn't work. It's the XPOs that cause the issue. You can clean it up in TFS technically, but it's a pain to maintain. Typically I advise people that there's no good reason to develop in multiple models in the same environment - why not put it in 1 model? And if it's multiple models, it should be a good idea to develop it in different environments anyway, which avoids the issue entirely. It's a mindset, but it's possible your reasons to use multiple models are not good, or you need to look at different dev environments, which should work if your reasons for different models work :-)

      Another avenue i haven't looked into further, is developing the models in different layers, then fixing the layer issue in the build (ie put the models back in the same layer).

      Delete
    2. An upgrade from 09 to 12 while trying to isolate a huge module that may be removed is the reason for the dev in the same environment. Also a good deal of code built on top of the module. Now how would I merge two models in TFS then once this has already started? Just physically move the files in TFS? Might have made a problem for myself...

      Delete
    3. Yes if you move the files in TFS, you will also move the history on the object with it.

      Delete
  8. This may be a stupid question but how do i check in models supplied by ISV's into TFS without having a licence key to log into the layer itself?

    ReplyDelete
    Replies
    1. Don't add the code, add the model. My activities have a step to import a model - for which you don't need layer codes.

      Delete