Thursday, June 14, 2018

Beating the Drum on Packages and Models

Even though I work on the product side these days and am crazy busy, I keep a close eye on the community out there. I have RSS feeds going for many blogs, I scan LinkedIn, I watch our insider Yammer groups, and I even follow an RSS feed of the official Community forum showing me all posts for AX/365 (yeah this is a lot but I scan through the titles quickly for specific things).

It's clear to me more and more people are moving onto AX7.x and as partners and ISVs understand the new world of X++ v7.0 things run smoother, bug reports become more useful, etc. That said, there is obviously still an influx of new people (partners, ISVs and customers) who are just now starting to learn the new paradigms. So, it doesn't hurt to go through some details again - this time I will focus on specific pain points I've seen people struggle with, especially when upgrading from AX2012.


#1 Rule of Thumb: A package is a mini-modelstore in and of itself, with its own set of layers and models!
I can't stress this enough. The standard application code is split up into multiple packages. If you're upgrading your over-layering code which is neatly contained in 1 model - it will get split up all over the place. And it will be split into models in existing packages as it needs to over-layer something in that specific package. Currently the code upgrade will not create an entirely new package for you. Do not expect to see a package with your name on it, but rather expect a model in multiple packages with your name on it. If you have a 2012 model called "Joris" then you may end up with an "ApplicationSuite\Application Suite Joris" and a "GeneralLedger\General Ledger Joris" folder. You will NOT see a package "Joris" in the root.
If you over-layer SalesTable - that over-layering can only be done in the package that contains SalesTable (Application Suite). (Note: you can EXTEND from any other package!) If you over-layer LedgerJournal - that over-layering can only be done in the package that contains LedgerJournal (GeneralLedger). Keep in mind over-layering of the standard code is completely disabled in application version 8.0 and upwards, so if you're upgrading you may need to go to 7.3 first to "buy time" to move things to pure extensions.


#2 Without a successful compile of the FULL package, you have nothing
When you get a new development VM, there's an application suite (package) present, and it has been compiled (each package is a unit of compilation - i.e. it translates to an assembly DLL for that package). If I now add a new class to that package which has a compile error in it...
1) Build/rebuild from a project/solution is ALWAYS an incremental compile on the assembly. This means in the class with error example, the application suite will be there but my new class won't, as the incremental compile wasn't able to compile and add my new stuff.
2) Full build from the Dynamics 365 menu is ALWAYS a full assembly compile. This means in the class with error example, the DLL will be completely recompiled. Now, the Visual Studio tools keep a backup of the DLL and in case of error, put it back. So in this case essentially nothing will happen - the DLL will be the exact same as before the compile since the tooling will just put the backup copy back.
3) Be mindful of removing objects and understand that build/rebuild from a project is an incremental compile. When in doubt, do a full build of your package to ensure removed objects are gone and new objects are added! When you've moved your code in its own package by extending instead of over-layering in existing packages, these compiles won't take long. If you're still over-layering, compiling something like Application Suite will obviously take quite a bit longer.


#3 Packages consume each other using references, and these are references to the BINARY (compiled) package
When Application Suite uses LedgerJournal, it can only do so because it has a reference to the GeneralLedger package where LedgerJournal is defined. But, this is a reference like any normal .NET reference. Let's say I add a completely new package and call it CodeCrib. I add a new class called BlogPost. Now, this class shows in the AOT in Visual Studio. But if I try to use this class in some Application Suite over-layering, it gives me a red squiggly line and it won't compile. So, I need to add a reference... From the Dynamics 365 > Model Management > Model Parameters screen, select your model in the drop-down and click Next to go to the references page, check the reference you need (to our CodeCrib package). This will add an official reference to the CodeCrib assembly (FYI this is stored in the descriptor of the package needing the reference).
Now, adding the reference resolves the squiggly line in the editor. But it still DOESN'T COMPILE! What gives? Well, when you compile it's all about the binaries! Since I haven't compiled the CodeCrib package yet, the compiler can't load the assembly when compiling AppSuite which references it!
So when upgrading and you run a full compile, you'll get LOADS of errors. But keep in mind that given #2 - a base package may not have compiled completely because of just 1 error. That could result in hundreds of errors in another package that depends on it! This is the way it works.
Also note that these references have nothing to do with cross-references! Cross-references are used for the Visual Studio tooling, but the compiler NEVER USES cross-references for anything. In fact, it (optionally) creates the cross-reference, but it doesn't consume it.
So best tactic when upgrading is to review package dependencies, and just start with the small packages that have little or no dependencies. Get those to compile, then work your way up. Application Suite is usually the LAST package you want to fix - since it won't compile anyway until you get its dependencies to compile properly. Once you have your own extension package, that one will likely become the last package to fix/compile, since it will depend on all the other packages it's trying to extend.


#4 Extensions and references - one-way traffic
The code upgrade does some work on moving obvious over-layers into extensions. This is ongoing work and as LCS updates and platform updates roll out, you may notice it doing more (you'll be using code upgrade moving between minor application upgrades: 7.1, 7.2, 7.3 but not for just platform updates). Moving to extensions ultimately implies it should go into its own package. But moving something to an extension means you need a reference to use it, which may not be possible since references are one way.
Let's say we have a new field on table SalesTable. And we have some over-layering code in SalesTableType class that uses this field. Now, if we move our new field to a table extension of SalesTable and we put that extension in a new standalone package we created, then the over-layering in SalesTableType won't find it. Now, to extend SalesTable in our package (let's call the package CodeCrib), we have to reference AppSuite. For our over-layering in AppSuite to reference our table extension, we need a reference to package CodeCrib. Unfortunately that would create a circular reference which you can't do. So there are two options: rework the over-layering in SalesTableType into extension as well (which ultimately you need to do), or keep the SalesTable extension in the AppSuite package for now, and move it later when you're ready to refactor the other over-layering.
For this reason the code upgrade, when it does create extensions, will put extensions in the original package where the over-layer existed - just so that you don't have reference issues to worry about while upgrading. You can then move a whole bunch of objects in their own package when you're ready.


#5 Conflict resolution
The code upgrade creates a conflict project for overlayering code. Although this is handy, it's important to remember that if you have code in MULTIPLE layers (let's say, you have sys code over-layered in VAR, and that VAR over-layered in CUS), you want to fix each layer independently, and work your way up. As you rework a layer, you may move some code around. Note that doing that may cause NEW CONFLICTS in a higher layer. Because of this, I would encourage you to create your own conflict project for each layer once you're done with a lower one to make sure no new objects have conflicts due to some refactoring you may have done in a lower layer. You can do this from the Dynamics 365 menu, under Add-Ins, Create project from conflicts.
Bottom line: when refactoring code in a lower layer, YOU could be creating new conflicts in a higher layer. This is not a bad thing, just keep it in mind and make sure to check for any new conflicting objects when using customizations in multiple layers. Note that if an object was already in a conflict project, any new conflicts in the same object will just show up when you open the designer for that object. You're more interested in new objects.
The add-in for creating a project can also be useful when you're done upgrading, just to go through each custom model to make sure nothing was left behind and you're really done.


#5 Binaries versus code
When you deploy a deployable package, it only contains binaries. There is no X++ code, only DLLs (and some other related artifacts needed). This means if you put a deployable package on a machine with Visual Studio, you will not see the code for the binaries you deployed. When you run the AOS of course you will run the binaries including your customizations. The larger issue I've seen is when you have overlayering code, let's say on Application Suite. When you deploy a package, you would be replacing the application suite compiled code (binaries). But the code on the machine is still standard application suite, without your custom code. So, after deploying, the AOS is running fine with your code. However, if you now open Visual Studio and run a compile on application suite you are replacing the application suite binaries (i.e. replacing the binaries you deployed via the deployable package). Since you're compiling the app suite without your custom code (since it wasn't deployed there) you are 'effectively' removing your customizations.


For all the talk about layers here, keep in mind that the existing packages are sealed in application version 8.0 and above - meaning you won't be allowed to over-layer any code in them and will be required to move to extensions. As such, when upgrading code you should consider the effort involved in upgrading all the code as-is, upgrading it by going through ALL the code (conflicting or not) and moving into extension. You still have the option to upgrade to 7.3 which has the application suite package still overlayerable (I just invented a new word).

And finally, if you're new to Dynamics 365 but have 2012 experience, I encourage you to read my Design,Compile,Run blog post series. It's hard to believe those articles are already more than 2 years old at this point...

Wednesday, June 13, 2018

Accidental Code Extensions

Ok, I'll preface this by saying I'm very much aware that the standard X++ code in platform and application has this issue too. Thanks for letting me know :-) But as the saying goes: do as I say - not as I do...

With that out of the way... Going back in time, the 7.0 X++ language supported extension methods like C#. You create a new class with any name, but ending in _Extension. Then, you can add a new public static method, and the first parameter is the object you're extending.
So for example:

static class MyExtensions_Extension
{
    public static void foo(PurchTable _purchTable)
    {
    }

    public static void bar(SalesLineType _salesLineType)
    {
    }
}
This class adds extension method foo() to the PurchTable table and method bar() to the SalesLineType class. This feature is now less used due to the [ExtensionOf()] "augmentation" class paradigm where you can have instance method, add member variables, access protected members, etc. However, the original extension method feature still exists, and in fact many people accidentally use it!

The issue happens when adding both extension methods and event handlers in the same class. In theory this sounds great - you have all your extensions in one place. For example:

static class MyPurchTableExtensions_Extension
{
    public void foo(PurchTable _purchTable)
    {
    }

    [DataEventHandler(tableStr(PurchTable), DataEventType::Inserting)]
    public static void HandlePurchInserting(Common sender, DataEventArgs e)
    {
    }
}

This works as expected - a new method foo() is added to PurchTable, and you're handling the inserting event. However, the unintended consequence is that you are ALSO adding a new method HandlerPurchInserting(DataEventArgs e) on the Common object! The compiler does not discriminate the fact that you have a handler attribute on that method. All it sees is you're adding a static method in an _extension class, with one or more arguments.


So... How many methods have you accidentally added to Common, XppPrePostArgs, FormRun or FormControl? :-)

Monday, November 27, 2017

Installing Hotfixes - Prepare vs Apply

Quick note on installing hotfixes and common questions around the usage of the prepare option. As I usually like to do, let's start with some background information.

I recommend a previous post I made on the topic of updates, upgrades and hotfixes.

When it comes to hotfixes, the platform is serviced as a whole. There aren't many hotfixes for platform since it is updated monthly anyway, but when there is it's cumulative and you install the whole thing at once as an update. Even though you get the source you don't worry about it going into source control or building and compiling it, because essentially you have the binaries already in the update which is ready to deploy to other non-dev environments, and you always get the whole source anyway - as opposed to deltas of just changed objects.
For application hotfixes however, you can get individual fixes for a specific KB you want. This will change when the application sealing comes around next year, but for now you have the option. Since all VMs come with a specific version of the application, any hotfixes you install will have to be synchronized to any place where the application may be compiled - so all dev boxes, build box etc. And of course when I say "synchronized" I mean the code for these hotfixes should go into source control.

Now, this way of doing things is slightly awkward. You get a base version of the application on the VMs, and any changes are in source control. This is nice and efficient because otherwise you'd have tens of thousands of source files in source control if you were to check-in the whole application. But from a source control perspective, this is weird. When you apply a hotfix and check in the changed object, source control will consider this a "new" object ("add") since you just added it to source control. But of course you technically already had it to begin with. What this means is if you were to "rollback" (undo) the add from source control - the opposite of an add is... delete. So if all dev boxes synchronized the hotfix (as an "add") and you rollback, all dev boxes when synching will say... hmm, I added this file and now it's gone, so let me delete it. As a result, you no longer have the object at all.

For this specific reason, the "prepare" option was added to the hotfix installation. Essentially, this will look at all objects in your hotfix, and add any of those objects to source control if they haven't already been. That way, if you want to rollback the hotfix, you have at least checked in the original version of the object (from before the hotfix) and you can roll back to that. Otherwise, you're rolling back to a delete (opposing the "add"). Now, a few additional pieces of information are important here:

1. An object could already have been added to source control. Because you did manually, because you had a previous hotfix on it, etc. This means that the number of objects in a pending changeset for a prepare step could be LESS than the number of objects in a pending changeset of a hotfix. A hotfix could technically also be adding objects, and new objects of course don't exist before applying the hotfix so they also wouldn't be in the prepare step.
2. After doing the prepare option, you have to check-in the pending changes. This is simple source control logic but many people forget this. You want to check-in the original versions of the objects prior to applying the hotfix. If you don't check it in before hitting apply, you won't have that base version! Simple as that. (in PU12 a dialog box was added after doing prepare that tells the developer to make sure to check-in prior to hitting apply).


Of course this also begs the question of how to recover if you did not prepare. Or if you did prepare but forgot to check-in before hitting apply. Truly, there is no easy way to recover from this. Ultimately, you need to get the original code from another VM that wasn't affected. So, consider these points:

- Did the hotfix go into source control and was it synchronized to other VMs as well? (i.e. this means removing the hotfix would affect multiple machines)
- Was it just an issue on your dev box, for example you hit apply right after prepare but haven't checked in anything (i.e. this means it's just your VM that needs to be fixed).

If the scope is just your machine, consider just getting a new VM. If you did check-in but no other machine has synchronized yet, you can rollback from source control (synchronize on other machines will see the add+delete which cancel out - VS will assume you didn't have the file since you didn't synchronize the "add", so it won't try to delete it) and you just need to worry about your machine.
Options for recovering all come down to getting the code from an unaffected VM (either deploy one temporarily, get a VHD, or with some luck someone else may have a VM that hasn't synced yet). The option is then to either manually copy the code - or potentially add these unaffected objects to source control from there (thus, creating your own baseline copy like prepare would have done).

Tuesday, October 31, 2017

PSA: AX7 Build Failure on Generate Packages / Model Export

More and more customers are seeing an error in the "Generate Packages" build step on their AX7 automated builds. The build shows as "Partially Succeeded" and the step that generates packages shows a problem.
The following error is shown in the build summary:


Error generating deployable packages: Error:
Unexpected exit code from model export: 1
At C:\DynamicsSDK\GeneratePackage.ps1:523 char:5


And going into the logs, the following details are shown:


- Foundation Upgrade: Exporting model source...
- Command: J:\AosService\PackagesLocalDirectory\Bin\ModelUtil.exe -export -metadatastorepath="J:\AosService\PackagesLocalDirectory" -modelname="Foundation Upgrade" -outputpath="C:\DynamicsSDK\VSOAgent\_work\1\Packages\Source"
Model Foundation Upgrade was not found in the specified Metadata Store
- Foundation Upgrade: Model export completed with exit code: 1
- Exception thrown at C:\DynamicsSDK\GeneratePackage.ps1:216: throw "Error: Unexpected exit code from model export: $ModelUtilExitCode"
System.Management.Automation.RuntimeException: Error: Unexpected exit code from model export: 1
Error generating deployable packages: Error: Unexpected exit code from model export: 1 At C:\DynamicsSDK\GeneratePackage.ps1:523 char:5


The automated build has an optional (turned on by default) to not just create the deployable package but also produce an export of all the models it built. The way this is done is it looks at any descriptor files for any models in source control, and those are the ones being exported (since those are also models for which their containing package are built).
That said, it appears many customers add the whole Descriptor folder to source control when over-layering existing packages. In itself this doesn't matter since the only extra work this creates is the model exports, it doesn't add any extra compile time since any one of the descriptors would cause the whole package to recompile.

The problem lies in the "Foundation Upgrade" model. This is an old artifact and this model is actually "Disabled" (a flag in the descriptor file). As such, our metadata APIs ignore this model - you will notice even though the descriptor and model are in your packages folder, this model is not showing up in the AOT. So, when the build tries to export this model, the model utility asks the metadata API for the model, and the API says it doesn't exist...

The fix of course is easy. The build scripts will be updated to double check the flag and not try to export disabled models. For customers or partners running into this issue today the answer is also easy - just remove the "foundation upgrade.xml" file from your source control. Note that this will trigger the deletion of that file on any dev boxes as well, but that is not an issue since this model isn't loaded or used anyway. A conversation has also started to see if this model should be downright removed in the next application release.

Saturday, October 21, 2017

Cross-Post: Pointing Build Definitions To Specific VMs (agents)

Dear readership,

Having a personal blog but also having opportunities to write on official blogs and documentation has its pros and cons. As I now have ownership of the AX Dev ALM blog on MSDN I will have the challenge of deciding when to post where regarding ALM related topics. I realize many people read this blog for some of the ALM related topics, so I figured I would cross-post here. I encourage you to follow the AxDevALM blog directly as it's not my intention to keep cross-posting everything. Perhaps some PSA stuff on the official blog and in-depth or example stuff here, we'll see.

For now, here's the link: Pointing Build Definitions To Specific VMs (agents).

And yes, I realize we're missing a lot of documentation on docs.microsoft.com regarding the build process. Posting there is a much lengthier and more stringent process, and blogging is just an easy, quick way to put something out there. The ALM blog is quick and painless, but it's official. Here, I can say (almost) anything I want :-)


I am working on other blog posts (on both blogs) so keep an eye out for those!

Thursday, June 22, 2017

Upgrades, Updates and Hotfixes in AX7

In Dynamics 365 for Operations (affectionately known as AX7), there is a clear distinction between binary hotfixes and metadata hotfixes (in LCS shown as "binary updates" and "X++ updates"). The product also has monthly platform releases and as of this writing a twice a year application release. I wanted to take a bit of time to explain the details of each and the distinction between them.

To provide context for this post, it's important to note that the product is split in two parts:
- Platform: kernel things, and some X++ code that are lower-level; things like batch framework, number sequences, print management, user info, etc.
- Application: The actual business logic. Customers, sales orders, production module, etc. Also some binary things like financial reporting (MR) and the SSRS framework for AX.


Platform Releases
After the release of Platform Update 4 (in February 2017), the platform now releases a monthly update. The platform consists of both the lower-level kernel binaries, as well as some X++ packages. However, the platform packages are "sealed" from over-layering, so even though you do get updated X++ code on your development boxes, you don't actually compile or customize these. You can extend them, of course. Each platform release contains any hotfix on the last platform, plus any new features (so they are cumulative - no need to install each one individually, just take the latest). Since this is just platform, the impact on the functionality is extremely minimal. Additionally, any code changes made by developers at Microsoft is checked for breaking changes (which are not allowed). This means the database schema cannot be changed (for the most part) and all APIs (method signatures etc.) should remain the same. This ensures any extensions won't break - the expectation is they don't need to be recompiled (i.e. you update platform, but no need to recompile/redeploy your extension packages). There are also backward compatibility checks to make sure older version of the application can run on newer versions of the platform.
In reality this all means that you can compile any code you have on an old platform, and deploy it on an environment with a newer platform - because the runtime is backwards compatible. As a result, many ISVs will develop and build their code on a "common" platform (for example, the fall release 2016 was released on Platform Update 3, and customers can install it on a newer platform (say, platform 7).
This also means that there are no individual hotfixes to pick. First, unless a really nasty issues appears, bugfixes will just come in the next month's update. But if there are critical fixes, they will be cumulative - meaning each new hotfix is just a newer build of the same platform; get the latest and you have all hotfixes.

Application Releases
The application actually consists of a few different things. Binaries and metadata. Application binaries are considered things like MR, SSRS components for AX, etc. Metadata is considered all the X++ code related to the application (sales orders, customers, etc.).
An application release is considered a major or minor version change. For example, the original version of AX7 had application 7.0. The fall release (2016) was 7.1. Spring release in 2017 is 7.2. Note that this is different than the platform - the platform will be backwards compatible, so there's no major or minor release change. An application release then also means it's both the binaries and the metadata combined as one release.


Now, how do you consume these updates and upgrades?


Application Metadata Hotfixes
Since as of this writing (2017!) you can still over-layer the application code, it means metadata hotfixes come in the form of X++ hotfixes. You have to install the hotfix on a development box, which fixes the existing code - and then you have to build and create a deployable package to actually consume the hotfix somewhere. You can download hotfixes from issue search of the "X++ updates" tile on an environment's page (at the bottom). Since these are just hotfixes, they have no impact on the actual application release you are running. You just have your original release, plus some hotfixes. Since you add those to source control to get built by the build VM, you only install them once and other developers will just synchronize them onto their boxes from source control.

Application Binary Hotfixes
Since the application is split in metadata as well as binaries, as explained above, you also need binary hotfixes next to the metadata hotfixes. These will be things like hotfixes for MR for example. These can be download from issue search, or also with the next update described. This will change once the application is sealed from over-layering.

Platform Updates
Platform updates effectively update your platform as described in the platform release above. A platform update can be download from the global asset library and deployed.

Binary Updates
On an environment page, there is also a tile called "Binary Updates". On environments of platform update 4 or newer, this effectively lets you download the latest version of the platform, as well as any application binary hotfixes. Since the tile is a combination of application binary hotfixes as well as platform updates, just installing a platform update from the global asset library (as described above) will not necessarily set the tile counter to 0 (zero) since there may still be applicable application binary hotfixes.

Application Upgrades
As of this writing, an application version is released twice per year (typically spring and fall releases). Since this changes the application functionality, it requires a code upgrade, full testing of the application and customizations, and a data upgrade.



Summary

WhatWhere to getContentsHow to DeployCode Upgrade RequiredData Upgrade required
X++ UpdatesEnvironment page
Issue Search
SCDPP package with X++ code changesInstall on Dev box, build/package, deployNO, but potential code conflict/merging required for over-layering codeNO
Binary UpdatesEnvironment pagePlatform Update (code+binaries)
Application Binary Hotfixes
LCS, manual runbookNONO
Platform UpdatesBinary Updates
Global Asset Library
Platform Update (code+binaries)LCS, manual runbookNONO
Application ReleaseLCS DeploymentEverything - application, platformRedeploy Environments (transparent for production environments)YESYES

Hope you like the retro HTML table styling.

Friday, March 31, 2017

Dynamics 365 for Operation Application Extensibility

At the 2017 Technical Conference, Microsoft announced the journey to support more customizations via extensions and will gradually remove the ability to over-layer the Application Suite, similar to what was done with the platform and foundation. This is referred to as application sealing.
This effort will come in waves and as we speak there's a lot of work being done at Microsoft to make the application more extensible - some new extension features but also a lot of refactoring, extension points, etc.

Please read the announcement as well as the steps in this journey on the Insiders Tips blog on the Dynamics Communities website.