Thursday, February 28, 2013

How We Manage Development - Organizing TFS

In the first post, I presented an overview of how we have architected our development infrastructure to support multiple clients. In this article, we will look at how we organize TFS to support our projects.

Team Foundation Server contains a lot of features to support project management. We have chosen to keep Team Foundation Server a developer-only environment, and manage the customer-facing project management somewhere else using SharePoint. In fact, even our business consultants and project managers do not have access or rarely see this magical piece of software we call TFS. (I bet you some are reading these articles to find out!)
There are several reasons for this, and these are choices we have made that I'm sure not everyone agrees with.
- Developers don't like to do bookkeeping. We want to make sure the system is efficient and only contains information that is relevant to be read or updated by developers.
- We need a filter between the development queue and the user/consultant audience. Issues can be logged elsewhere and investigated. AX is very data and parameter driven so a big portion of perceived "bugs" end up to be data or setup related issues. If anyone can log issues or bugs in TFS and assign them to a developer, the developers will quickly start ignoring the lists and email alerts because there are too many of them. TFS is actual development work only, any support issues or investigative tasks for developers/technical consultants are outside of our TFS scope.
- Another reason to filter is project scope and budget. Being agile is hot these days, but there is a timeline and budget so someone needs to manage what is approved for the developers to work on. Only what is approved gets entered in TFS, so that TFS is the actual development queue. That works well, whether you go waterfall or agile.
- Finally, access to TFS means access to a lot of data. There are security features in TFS, of course, but they are nothing like XDS for example. This means check-in comments, bug descriptions, everything would be visible. There are probably ways to get this setup correctly, but for now it's an added bonus of not having to deal with that setup.

Each functional design gets its own requirement work item in TFS. Underneath, as child workitems, we add the development "tasks". Initially, we just add one "initial development" task. We don't actually split out pieces of the FDD, although we easily could i guess. When we get changes, bugs, basically any change after the initial development, we add another sub-task we can track. So, for example, the tasks could look like this in TFS:



This allows us to track any changes after the initial development/release of the code for the FDD. We can use this as a metric for how good we are doing as a development group (number of bugs), how many change requests we get (perhaps a good measure of the quality of the FDD writing), etc. It also allows us to make sure we don't lose track of loose ends in any development we do. When a change is agreed upon, the PM can track our TFS number for that change. If the PM doesn't have a tracking number, it means we never got the bug report.
It is typically the job of the project's development lead to enter and maintain these tasks, based on emails, status calls with PMs, etc. I will explain more about this in the next article.

On to the touchy topic of branching. There are numerous ways to do this, all have benefits and drawbacks. If you want to know all about branching and the different strategies, you can check out the branching guide written by the Microsoft Visual Studio ALM Rangers. It is very comprehensive but an interesting read. There is no one way of doing this, and what we chose to adopt is loosely based on what the rangers call the "code promotion" plan.



Depending on the client we may throw a TEST branch in between MAIN and UAT. We have played with this a lot and tried different strategies on new projects. Overall this one seems to work best, assuming you can simplify it this way. Having another TEST in between may be necessary depending on the complexity of the implementation (factors such as client developers doing work on their systems as well - needing a merge and integrated test environment, or a client that actually understands and likes the idea of a pure UAT - those are rare).
One could argue that UAT should always be moved as a whole into your production environment. And yes, that is how it SHOULD be, but few clients and consultant understand and want that. But, for that very reason, and due to the diversity of our clients' wants and needs, we decided on newer projects to make our branches a bit more generic, like this:



This allows us to be more consistent in TFS across projects, while allowing a client to apply these layers/models as they see fit. It also allows us to change strategies as far as different environment copies at a client, and which code to apply where. All without being stuck with defining names such as "UAT" or "PROD" in our branch names.

So, the only branch that is actually linked to Dynamics AX is the MAIN branch. Our shared AOS works against that branch, and that is it. Anything beyond that is ALL done in Visual Studio Explorer. We move code between branches using Visual Studio. We do NOT make direct changes in any other branch but MAIN. Moving code through branch merging has a lot of benefits.

1) We can merge individual changes forward, such as FDD03, but leave FDD04 in MAIN so it is not moved. This requires developers to ALWAYS do check-ins per FDD, never check-in code for multiple FDDs at the same time. That is critical, but makes sense - you're working on a given task. When you go work on something else (done or not), you check in the code you're putting aside, and check out your next task.
2) By moving individual changes, TFS will automatically know any conflicts that arise. For example, FDD02 may add a field to a table, and the next FDD03 may also add a field to that same table. But both fields are in the same XPO in TFS - the table's XPO. Now, TFS knows which change you made when... so whether we want to only move FDD02, or only FDD03, TFS will know what code changes belongs to which. (more on this in the next article) This removes the messy, manual step of merging XPOs on import.
3) The code move in itself becomes a change as well. For clients requiring strict documentation for code moves (SOx for example), the TFS tracking can show that in the code move, no code was changed (ie proving that UAT and PROD are the same code). We also get time stamps of the code move, user IDs, etc. Perfect tracking, for free.
4) With the code move itself being a change set, we can also easily REVERT that change if needed.
5) It now becomes easy to see which pieces of code have NOT been moved yet. Just look for pending merges between branches.

So, I can look at my history of changes in MAIN, and inquire into the status of an individual change. In the first screenshot, you see a changeset that was merged all the way into RELEASE. My mouse is hovered over the second branch (test) and you can see details. The second screenshot shows a change that has not been moved into RELEASE yet.




Obviously you can also get lists of all pending changes etc. We actually have some custom reports in that area, but you can get that information out of the box as well.

So, for any of you that have played with this and branching, the question looming is: how can you merge all changes for one particular FDD when there are multiple people working on multiple FDDs. TFS does not have a feature to "branch by work item" (if you check in code associating it to a work item, one could in theory decide to move all changes associated with that work item, right?). Well, the easiest way we found is to make sure the check-in comment is ALWAYS preceded with the FDD number. So, when I need to merge something from MAIN to TEST, I just pick all the changesets with that FDD in the title:



Yes, you may need several merges if there are a lot of developers doing a lot of changes for a lot of FDDs. However, it forces you to stay with it, and it actually gets better: if you merge 5 changesets for the same FDD at the same time, the next merge further down to the next branch will only be for the 1 merge-changeset (which contains all 5). Also consider the alternative of you moving the XPOs manually and having to strip out all the unrelated code during the import. After more than 2 years of using TFS heavily, I know what I prefer :-) (TFS, FYI).

Once code is merged, all that needs to happen is a right-click by the lead developer to start a new build - no further action required. This will spin up TFS build using our workflow steps, which imports all the code, compiles, and produces the layer (ax 2009) or model (ax 2012). More details on this in the next article.



Once the build is completed (successfully or not), we get an alert via email. If the build was successful, the model or layer will be on a shared folder - ready to be shipped to the client for installation, with certainty that it compiles. We actually have a custom build report we can then run which, depending on the client's needs, will show which FDDs and which tasks/bugs were fixed in this release, potentially a list of the changed objects in the build, etc. Without the custom report, here's what standard TFS will give you:



As you can tell by the run time (almost 3 hours), this is AX 2012 :-)

This has become a lengthy post, but hopefully interesting. Next article, I will show you some specific examples (including some conflicts in merging etc) - call it a "day in the life of a Sikich AX developer"...

34 comments:

  1. Thanks for the excellent sharing. I'd been playing with TFS since it is available on the cloud. Your kind sharing give me a much better idea of how I could use it in real project dev. situation.

    ReplyDelete
  2. Hi Joris. Great articles.

    I was wondering how you manage building multiple branches and working out what is included in them.

    For example you have done some development in DEV and merged it into TEST then built TEST and tested the code changes. Now you want to move those changes to LIVE, how can you be sure what you tested is what you are shipping if you have another merge to perform? Or have a missed something here?

    ReplyDelete
    Replies
    1. Good question, and I may not have explained well enough.

      Two answers to your question:

      1. Technically, what you test should be exactly what you move. In any other product this means moving the binary. For AX that would mean once everything in your UAT is approved and tested, you move THE WHOLE MODEL STORE to production. That is how it should be, but I don't think there's a lot of people who do it that way. If you do a full move of the same individual model containing all the customizations, you're half way there. But technically you're not moving exactly what you tested.

      2. So what we do in a case where the customer cannot be convinced to move the whole UAT. We do move one FDD at a time to production, if we have to (and unfortunately this happens a lot). But at least using TFS, there's a few positives:
      a) any merge conflicts will be pointed out or resolved by TFS. So no accidental merge issues
      b) we can do a compare in TFS between two branches to find out the differences. The only difference we see should be the things that weren't moved to production.

      But to your point, one should move the whole model store from UAT to PROD, that's the only 100% correct way. Any and all other ways your integrated testing in UAT was useless, as issues could have been introduced by splitting the code.

      Delete
  3. Joris, what about planning, do you know if there are any planning tools in TFS?
    Otherwise we need to chat on skype or so...

    Grtz,
    Philippe

    ReplyDelete
    Replies
    1. I'm not entirely sure what you mean by planning? If you keep your work items and estimates in TFS, you can have "hot" links between TFS and MS Project or Excel. You can manage the work items from that excel list or MS Project file, and push updates back to TFS (there's a TFS plugin for both). So that would allow you to use Ms Project to do scheduling or something. Not sure if that's what you were asking for?

      Delete
    2. OK, I wasn't sure if there were any expliciet planning functionalities in TFS, but your solutions seems good!

      Delete
  4. Joris, this may be a bit off topic, but have you had any problems with syncing labels from TFS into AX? Or even checking out a label file?
    It seems to crash the AX client on 64 bit Win8 / Server 2012 when running from those OS. Putting the AX client on a Win7 32 bit seems to work though, but I'd prefer not to have to spin up X numbers of Win7 VM's.

    ReplyDelete
  5. Hello Joris.

    I would like to use the same solution to merge change from MAIN to TEST. But, is there a way to filter the changesets with the comment column ?

    We have 30 developers working with TFS. Find all changesets with same comment is little bit boring as we can't organize the comment column in the merge wizard.

    Do you have to read all the lines ?

    ReplyDelete
    Replies
    1. Yes, out of the box you can't filter. However, there are some open source add-ons on CodePlex that allow you to merge changesets based on all sort of interesting things - for example one that allows you to merge by workitem (so it checks which changesets are tied to the workitems), etc. So the APIs from TFS are there, and some people have used those to create cool add-ons.
      But yes, we merge them individually - assuming we keep up the merging of fixes and don't wildely check-in, it hasn't really presented many challenges.

      If you have 30 developers all working on the same AX codebase that is quite the scenario where I would think merging changesets is the least of your problems :-)

      Delete
    2. Thank you Joris for your answer. I was not able to find any add-on to do the "merge by workitem". Do you have one to provide ?

      Delete
    3. This is one I found: http://mergeworkitems.codeplex.com/

      Delete
  6. Hello,

    Great post !

    I have one question though: Reading the ALM Rangers branching guide, it says the Code Promotion plan seems "out-dated". Is it possible to branch for release isolation or development isolation in AX? Have you tried these? What makes it better or worse than code promotion branching?

    Thanks!
    Nicolas

    ReplyDelete
    Replies
    1. Ultimately AX is code like any other. There's probably no reason why any branch plan wouldn't work. But from experience, the issues are culture in the industry, not technology or branch plans.
      Since I wrote this article we've actually moved onto a model where we code promote into a main branch, but only have 1 release branch that goes to our clients. I guess it's somewhat a hybrid between release and code promotion branching. We do code promotion internally, but we only release to a client from 1 branch - forcing them promote full models between their environments, as opposed to asking to promote certain customizations.

      Delete
    2. OK, that's what I had in mind; Thanks!

      Delete
  7. Great Article!

    We have been working with AX 2012 R2 and now R3 and are a Dynamics AX customer; not a software shop. We've implemented a similar branching structure as you've mentioned. The only real difference, and where I'd like your thoughts, is in the merging/sequence process. Let me explain:

    We have been in the phase for a few months now where we have about 10 developers all working on different FDDs with many of them overlapping/using shared objects. This means dev 1 is working on several objects that dev 2 will also be modifying.

    Let's use the image above as an example: say DEV 1 checks in 10447 with incomplete work. DEV 2 however has finished their code and has checked in 10468. If there is an overlap of their objects and DEV 1 checks in code not yet complete, our QC environments then are not testing completed code and pick up 'false' defects due to the incomplete code being in place. BTW, completed code in this case means that it's ready for QC.

    The only way we found around this was to basically institute a policy that only completed code is actually checked into the DEV branch. Then we merge that branch to QC which is what the QC Build machine pulls from. If a developer has incomplete code that they want to store for any reason, we are having them use TFS shelvesets.

    Is there something that I'm failing to understand in our situation that you could suggest a correction/alternative. We have a pretty formal promotion process and the QC requirement is that they need to ensure that they are testing completed code only so they can be assured that 'incomplete' code is not impacting any of the completed items.

    Thanks,

    ReplyDelete
    Replies
    1. We've changed our branching structure since I wrote this to be more "general" development. We have DEV <= MAIN => RELEASE. Main is still internal and is where QC happens. So basically, using one of the branches for QC means you can safely merge one feature without the other, and not have any unfinished code in your QC branch (because you deal with it in the merge). Does that make sense?

      Delete
    2. It helps some but I'm still left with the question: Assume I've got two changesets checked into MAIN at the same time and have some overlapping of code (say a shared class). Also assume that the first changeset failed QC but the second changeset passed QC. How would you move one changeset forward to release while holding the other back and resubmitting to DEV if there is a shared class between them? Would we have to 'remove' the part of the shared class that was developed as part of the first changeset?

      Delete
    3. TFS handles that for you. If there's conflicts TFS will point those out... Sometimes it fixes them automatically (since it obviously knows exactly what each change does), other times it will just alert you of the two changes and ask you to tell it how you want to handle.
      I suggest you try it somewhere, it'll make sense once you see it.

      Delete
    4. That's not quite what I meant. I understand about conflicts from TFS perspective and how it can auto fix or require you to merge manually prior to checkin. What I'm not sure on is let's say you and I are the developers. You've checked the code in changeset 76 containing WHSShipConfirm. I start my checkin but TFS notifies me that the WHSShipConfirm class has been previously modified and that I need to manually resolve the conflict. I complete my checkin and get ChangeSet 77. During QC, they fail changeset 76 buy pass my changeset 77. However, if we promote changeset 77 forward into production it has 'partial' changes from changeset 76 due to that manual merge that I performed as part of the TFS conflict resolution. Since QC failed changeset 76 no piece of it is supposed to move forward. How do you resolve this type of scenario?

      Thanks again Joris for this blog and your responses...

      Delete
    5. Those conflicts also arise when you move code between branches... so having one branch you develop in and one branch you test in (and maybe another you release from) has the same effect.

      Delete
  8. With your DEV<=Main=>Release setup, do your developers all connect to the Dev branch in AX, and then check-in against work items in AX? What about the "initial development" task?

    Then, do they merge a changeset attached to a work item to the main branch for testing?

    If you have 3 unrelated FDDs checked-in to Main (changes 5, 6, 7). Users test in Main and 5/7 pass but 6 fails. Do you just merge changeset 5 & 7 into Release and rollback 6?

    This blog post, along with your new method seems like it needs to be a YouTube video for me to get my hands around it. Great post though. I'm trying to do this for a Sox setup, so it's a little harder for me to just jump and try it, then change something up when we realize what works better.

    ReplyDelete
  9. Hi Joris. Thank you for sharing your TFS setup. I have a question regarding your branch strategy when it comes to releasing hotfixes. Do you branch off for each release or use TFS labels in a single Release branch and only branch if a hotfix is needed for a particular release? Also, with the new DEV <= MAIN => RELEASE approach you adopted, do you use AX to check in code to DEV branch only and merge using TFS to Main and Release branches?

    ReplyDelete
  10. So yes, it's checking in code against DEV branch only, then merge using TFS.

    As for the release branching, I'm assuming you are talking about product development and not customer development. For customer development, those three branches are it, we never created any special branches (except perhaps for an upgrade project while you're still maintaining a live environment). For product development, there are a lot of strategies to go with. Generally you want to keep the number of branches as low as possible, but ultimately you're going to need release branches. I suggest review the ALM Ranger's branching guide: https://vsarbranchingguide.codeplex.com/ (check the downloads page).

    ReplyDelete
    Replies
    1. Hi Joris. Thank you for your reply. I have another merge question. Our dev team makes checkins using AX to Mainline var folder. Can we use TFS to merge changesets from Mainline's var folder to release branch's vap folder (in order to provide a vap layer hotfix)? Right now we import xpos into vap layer using AX and use the compare tool to bring required code into vap.model.

      Delete
    2. First, I question the usefulness of using P-layers in AX2012 now that you have models. It's a historical thing to do but in most cases it's not very useful.
      Secondly, how do you maintain that in TFS? The next release you literally delete all the objects, merge the old changesets into VAR and move the new changesets into VAP? and repeat later? This will get out of hand quickly. And again, questioning what the real value is of doing VAP models.

      Delete
  11. Hi Joris. I have an interesting situation that has arisen and I'm not sure if I'm overthinking this or not. We are running AX 2012 R3 RTM currently. However, we are in the process of upgrading to AX 2012 R3 CU9 and doing new development work on CU9 while still doing production support changes. Every developer has 2 VM dev boxes (one for R3 RTM and one for R3 CU9). There are two corresponding QC environments as well. All boxes are hooked to TFS.

    TFS is setup like the following:

    AX (TFS Project)
    - Release1
    - DEV (Branched to QC)
    - QC (Branched to Release)
    - Release (Branched to Release2/DEV to create initial Release2 branch at time of go 1st company go live.)
    - Release2
    - DEV (Branched to Release2/QC)
    - QC (Branched to Release2/Release)
    - Release

    The R3 RTM boxes are hooked to the corresponding Release1 branches while the R3 CU9 boxes are hooked to the Release2 branch. When we did the CU9 upgrade, a Release2 dev box was used that contained the latest R3 RTM production modelstore. Once the CU9 code merge was complete and the Release2/DEV branch was updated with the code merge; we rolled out CU9 and the imported the upgraded modelstore to the rest of the boxes hooked to Release2.

    My Dilemna:
    Currently, we have developers working production issues and fixing defects or adding new functionality to the Release1 development stream; standard QC builds are done and once approved go to production.

    Those same developers, take their approved Release1 code and then are manually importing their code to their CU9 boxes, performing code merge steps, and then checking it into Release2/DEV.

    This cycle will continue until we are eventually ready to upgrade the production environment to CU9 including the TFS code from the Release2/Release branch.

    However, using this pattern, are we introducing the chance for ID conflict issues to arise in production? Since the developers have new code going into the production R3 RTM modelstore but have to manually import into their CU9 boxes and check that into the Release2; won't the modelstore that we promote to production from the Release2 have ID issues?

    I know this was a long post....sorry about that. I'm hoping I'm overthinking/missing something.

    Thanks,
    Michael

    ReplyDelete
    Replies
    1. One clarification - the TFS structure looks like this:

      AX (TFS Project)
      - Release1
      --- DEV (Branched to QC)
      --- QC (Branched to Release)
      --- Release (Branched to Release2/DEV to create initial Release2 branch at time of go 1st company go live.)
      - Release2
      --- DEV (Branched to Release2/QC)
      --- QC (Branched to Release2/Release)
      --- Release

      Delete
    2. In the optimal scenario your released code freezes except for hotfixes. You then make hotfixes directly in your release branch, and branch/merge backwards into your DEV branch where you're working on release 2. In that case you don't QC on a QC branch but on a specific build of the branch your released code. You're probably already locked into the structure, but it looks like you essentially have no TFS branch relationship between release1 and release2, hence your having to manually port code over. It looks like you have a good branching structure/procedure for your released code so I encourage you to rethink your upgrade branching for next time :-)

      As for your question about object IDs, the answer is yes and no and maybe. For AX2012, the XPOs going into TFS have no IDs in them, which means you technically have the same issues today between your branches. So it all depends on how you deploy the code from TFS back into AX. Whatever your process is could be the same for your upgraded code process, unless you're doing something special for your release process today.

      Delete
    3. Thanks Joris for the quick reply. When it was just Release1 that we had, we would take the modelstore from production and import it to the QC build machine. We would then perform a TFS Synch on the QC Build machine to pull from the QC branch. Once QC verifies it, we promote by exporting the QC modelstore and promoting to production via the standard modelstore promotion process. However, how would that work now with the Release2 QC build machine?

      The production modelstore was originally imported into the Release2 build machine and then a CU9 upgrade performed. So I can't bring the production modelstore (with fixes/enhancements) back to the Release2 QC unless I want to re-perform a CU9 upgrade. If I don't do that though, then I run the risk of ID conflicts if I understand correctly. We were initially going to have several months difference between Release1/Release and the ultimate release of CU9 (Release2/Release) which means we would have a greater volume of production changes (more potential for ID conflicts). Thoughts? You also mention about rethinking the upgrade branching. Do you have any suggestions? If I need to make changes or restructure, now is probably the best time for me to do so to prevent this from happening again.

      Thanks,

      Delete
  12. Hi Joris, your articles are great as always. I have a question regarding implementing these TFS branches for different environments in the middle of an implementation where there is already a vast amount of developers that have checked in code in DEV, and a subset of that already moved to TEST (manual XPO move), and a subset of that already passed testing and moved to UAT (again, manually). How do you set up the branching when there is already parts of the code already scattered throughout the environments. Ideally, you want to have the branches set up on Day 1 of development, but I'm in a situation where I'm running into a client where they have a terrible code promotion process already established and I'm attempting to get them to a better process but I don't want to branch their current DEV state, as there is much not-ready-for-test code.

    ReplyDelete
    Replies
    1. Some sort of baselining is needed. Take for example the production code. Check that in. Then checkin all the dev code on top of it. TFS will only check-in anything that is different. You'll lose some granularity of check-ins, of course, but at least you'll some sort of delta to work with.
      Arguably, any other environments between prod and dev can be ignored - or known outstanding could can be moved to prod or just make sure it's in dev.

      Delete
    2. I'm not exactly sure how to do this. We're already checking things into TFS from a DEV environment and attaching the changesets to TFS work items. How do I get a baseline from another environment when TFS is hooked into DEV?

      Delete