Tuesday, October 31, 2017

PSA: AX7 Build Failure on Generate Packages / Model Export

More and more customers are seeing an error in the "Generate Packages" build step on their AX7 automated builds. The build shows as "Partially Succeeded" and the step that generates packages shows a problem.
The following error is shown in the build summary:

Error generating deployable packages: Error:
Unexpected exit code from model export: 1
At C:\DynamicsSDK\GeneratePackage.ps1:523 char:5

And going into the logs, the following details are shown:

- Foundation Upgrade: Exporting model source...
- Command: J:\AosService\PackagesLocalDirectory\Bin\ModelUtil.exe -export -metadatastorepath="J:\AosService\PackagesLocalDirectory" -modelname="Foundation Upgrade" -outputpath="C:\DynamicsSDK\VSOAgent\_work\1\Packages\Source"
Model Foundation Upgrade was not found in the specified Metadata Store
- Foundation Upgrade: Model export completed with exit code: 1
- Exception thrown at C:\DynamicsSDK\GeneratePackage.ps1:216: throw "Error: Unexpected exit code from model export: $ModelUtilExitCode"
System.Management.Automation.RuntimeException: Error: Unexpected exit code from model export: 1
Error generating deployable packages: Error: Unexpected exit code from model export: 1 At C:\DynamicsSDK\GeneratePackage.ps1:523 char:5

The automated build has an optional (turned on by default) to not just create the deployable package but also produce an export of all the models it built. The way this is done is it looks at any descriptor files for any models in source control, and those are the ones being exported (since those are also models for which their containing package are built).
That said, it appears many customers add the whole Descriptor folder to source control when over-layering existing packages. In itself this doesn't matter since the only extra work this creates is the model exports, it doesn't add any extra compile time since any one of the descriptors would cause the whole package to recompile.

The problem lies in the "Foundation Upgrade" model. This is an old artifact and this model is actually "Disabled" (a flag in the descriptor file). As such, our metadata APIs ignore this model - you will notice even though the descriptor and model are in your packages folder, this model is not showing up in the AOT. So, when the build tries to export this model, the model utility asks the metadata API for the model, and the API says it doesn't exist...

The fix of course is easy. The build scripts will be updated to double check the flag and not try to export disabled models. For customers or partners running into this issue today the answer is also easy - just remove the "foundation upgrade.xml" file from your source control. Note that this will trigger the deletion of that file on any dev boxes as well, but that is not an issue since this model isn't loaded or used anyway. A conversation has also started to see if this model should be downright removed in the next application release.

Saturday, October 21, 2017

Cross-Post: Pointing Build Definitions To Specific VMs (agents)

Dear readership,

Having a personal blog but also having opportunities to write on official blogs and documentation has its pros and cons. As I now have ownership of the AX Dev ALM blog on MSDN I will have the challenge of deciding when to post where regarding ALM related topics. I realize many people read this blog for some of the ALM related topics, so I figured I would cross-post here. I encourage you to follow the AxDevALM blog directly as it's not my intention to keep cross-posting everything. Perhaps some PSA stuff on the official blog and in-depth or example stuff here, we'll see.

For now, here's the link: Pointing Build Definitions To Specific VMs (agents).

And yes, I realize we're missing a lot of documentation on docs.microsoft.com regarding the build process. Posting there is a much lengthier and more stringent process, and blogging is just an easy, quick way to put something out there. The ALM blog is quick and painless, but it's official. Here, I can say (almost) anything I want :-)

I am working on other blog posts (on both blogs) so keep an eye out for those!

Thursday, June 22, 2017

Upgrades, Updates and Hotfixes in AX7

In Dynamics 365 for Operations (affectionately known as AX7), there is a clear distinction between binary hotfixes and metadata hotfixes (in LCS shown as "binary updates" and "X++ updates"). The product also has monthly platform releases and as of this writing a twice a year application release. I wanted to take a bit of time to explain the details of each and the distinction between them.

To provide context for this post, it's important to note that the product is split in two parts:
- Platform: kernel things, and some X++ code that are lower-level; things like batch framework, number sequences, print management, user info, etc.
- Application: The actual business logic. Customers, sales orders, production module, etc. Also some binary things like financial reporting (MR) and the SSRS framework for AX.

Platform Releases
After the release of Platform Update 4 (in February 2017), the platform now releases a monthly update. The platform consists of both the lower-level kernel binaries, as well as some X++ packages. However, the platform packages are "sealed" from over-layering, so even though you do get updated X++ code on your development boxes, you don't actually compile or customize these. You can extend them, of course. Each platform release contains any hotfix on the last platform, plus any new features (so they are cumulative - no need to install each one individually, just take the latest). Since this is just platform, the impact on the functionality is extremely minimal. Additionally, any code changes made by developers at Microsoft is checked for breaking changes (which are not allowed). This means the database schema cannot be changed (for the most part) and all APIs (method signatures etc.) should remain the same. This ensures any extensions won't break - the expectation is they don't need to be recompiled (i.e. you update platform, but no need to recompile/redeploy your extension packages). There are also backward compatibility checks to make sure older version of the application can run on newer versions of the platform.
In reality this all means that you can compile any code you have on an old platform, and deploy it on an environment with a newer platform - because the runtime is backwards compatible. As a result, many ISVs will develop and build their code on a "common" platform (for example, the fall release 2016 was released on Platform Update 3, and customers can install it on a newer platform (say, platform 7).
This also means that there are no individual hotfixes to pick. First, unless a really nasty issues appears, bugfixes will just come in the next month's update. But if there are critical fixes, they will be cumulative - meaning each new hotfix is just a newer build of the same platform; get the latest and you have all hotfixes.

Application Releases
The application actually consists of a few different things. Binaries and metadata. Application binaries are considered things like MR, SSRS components for AX, etc. Metadata is considered all the X++ code related to the application (sales orders, customers, etc.).
An application release is considered a major or minor version change. For example, the original version of AX7 had application 7.0. The fall release (2016) was 7.1. Spring release in 2017 is 7.2. Note that this is different than the platform - the platform will be backwards compatible, so there's no major or minor release change. An application release then also means it's both the binaries and the metadata combined as one release.

Now, how do you consume these updates and upgrades?

Application Metadata Hotfixes
Since as of this writing (2017!) you can still over-layer the application code, it means metadata hotfixes come in the form of X++ hotfixes. You have to install the hotfix on a development box, which fixes the existing code - and then you have to build and create a deployable package to actually consume the hotfix somewhere. You can download hotfixes from issue search of the "X++ updates" tile on an environment's page (at the bottom). Since these are just hotfixes, they have no impact on the actual application release you are running. You just have your original release, plus some hotfixes. Since you add those to source control to get built by the build VM, you only install them once and other developers will just synchronize them onto their boxes from source control.

Application Binary Hotfixes
Since the application is split in metadata as well as binaries, as explained above, you also need binary hotfixes next to the metadata hotfixes. These will be things like hotfixes for MR for example. These can be download from issue search, or also with the next update described. This will change once the application is sealed from over-layering.

Platform Updates
Platform updates effectively update your platform as described in the platform release above. A platform update can be download from the global asset library and deployed.

Binary Updates
On an environment page, there is also a tile called "Binary Updates". On environments of platform update 4 or newer, this effectively lets you download the latest version of the platform, as well as any application binary hotfixes. Since the tile is a combination of application binary hotfixes as well as platform updates, just installing a platform update from the global asset library (as described above) will not necessarily set the tile counter to 0 (zero) since there may still be applicable application binary hotfixes.

Application Upgrades
As of this writing, an application version is released twice per year (typically spring and fall releases). Since this changes the application functionality, it requires a code upgrade, full testing of the application and customizations, and a data upgrade.


WhatWhere to getContentsHow to DeployCode Upgrade RequiredData Upgrade required
X++ UpdatesEnvironment page
Issue Search
SCDPP package with X++ code changesInstall on Dev box, build/package, deployNO, but potential code conflict/merging required for over-layering codeNO
Binary UpdatesEnvironment pagePlatform Update (code+binaries)
Application Binary Hotfixes
LCS, manual runbookNONO
Platform UpdatesBinary Updates
Global Asset Library
Platform Update (code+binaries)LCS, manual runbookNONO
Application ReleaseLCS DeploymentEverything - application, platformRedeploy Environments (transparent for production environments)YESYES

Hope you like the retro HTML table styling.

Friday, March 31, 2017

Dynamics 365 for Operation Application Extensibility

At the 2017 Technical Conference, Microsoft announced the journey to support more customizations via extensions and will gradually remove the ability to over-layer the Application Suite, similar to what was done with the platform and foundation. This is referred to as application sealing.
This effort will come in waves and as we speak there's a lot of work being done at Microsoft to make the application more extensible - some new extension features but also a lot of refactoring, extension points, etc.

Please read the announcement as well as the steps in this journey on the Insiders Tips blog on the Dynamics Communities website.

Wednesday, November 2, 2016

Pointing Your New Build Definition to a Different Branch

This will be quick, I promise! Many people ask about branching and I promise some day I'll revisit my "how we do development" article to reflect my personal opinions on simple branch setups for AX7/Dynamics 365 for Operations (we really need a shorter conversational name, but I'll wait for the community to find something good and stick with SEO for now :-)

But, skipping the branching discussion for second, if you do use branches, how do you deal with builds? Well, that is really easy. You clone the existing definition, change the repository mapping and change the build task to point to the correct folder's project file!

Step 1: Clone the original build definition
Click on the ellipsis (...) next to the build definition's name in your build definition list, and click "Clone".

Step 2: Change the repository mapping
Cloning should copy the original definition and open it for editing. On the "Repository" tab, change the "Mappings" to point to your new Server Path for your branch. (this screenshot shows trunk/main still, change that to what you need).

Step 3: Change the project file location in the build tasks
On the "Build" tab, select the "Build the solution" task and edit the project location. I advise to use the browse feature by clicking the ellipsis, to make sure you don't mistype.

Step 4: Save & Run!
That's it! Make sure to save your changes, and on the save dialog give the new build definition a proper name. At this point you can create multiple build definitions for the branches. You can play with setting for example any dev branches as gated builds or nightly builds, change the build numbering to clearly distinguish between builds from different branches, etc. Go wild!

Cool. This must be the shortest post I've done in years! Happy building!

Monday, October 3, 2016

The Overall Concept of Extensions in AX 7

Unfortunately Dynamics AX 2012 also introduced a concept of extensions, which I blogged about here, but it's not quite the same thing. The extensions from AX 2012 have sort of transformed in what is now known as plug-ins. More on that in a future article.

Although I'm enthusiastic about extensions, and many people share my enthusiasm, I've consciously (and with a hard-to-gather dose of restraint) decided to blog about the paradigms around packages and the package split first. The reason is to frame extensions properly. Extensions are a way to add features and functionality to the existing application, yes, but they are more than just a new technology feature in AX. They are an ENABLER to make the new paradigms and the strategy around packages successful. Developers should strive to use extensions to drive a move towards APIs and packages, not just to avoid over-layering (although the two are closely related). The goal is to move into independent loosely-coupled packages, and to attain that goal over-layering should be avoided, yes.

So what are extensions? If you think about customizations and Partner and ISV solutions, for the most part they add features and functionality to the existing application. I.e. they extend it. For this to succeed, they must be able to add fields to existing tables, execute code when certain things in the standard application happen, etc. Traditionally, this is done by over-layering the standard objects and code, and adding any needed new features. With extensions, the goal is to allow adding these additional features without over-layering, and even extending them from different packages. This seems fairly straight-forward from a metadata perspective. Indeed, adding a field to a table was already very granular by layer and model in AX 2012. And adding a form to a control was possible at runtime using personalizations. From a code perspective, however, things seem much harder. In AX 2012, eventing was introduced. One could add a pre- or post-handler to a method without technically over-layering the method. However, if for example a table didn't define a modifiedfield or update method one had to over-layer the table first to add the method, then add a handler to it. This obviously defeats the purpose entirely. And even when one added a handler or even an entirely new method to an existing class, you were technically still over-layering the class itself in a way.

So yes, beyond the expected tooling around adding fields to tables, controls to forms, etc., there are concepts to facilitate code extensions as well. First and foremost, events are everywhere. Although the traditional way of overriding a base method on a table or form or form control still is possible, most of those methods which were semi-events, are now ACTUAL events. A table for example has an OnUpdating and OnUpdated event, which allows you to subscribe to a pre- and post-update event without creating any methods on the table itself. A form button control actually has events for a click event, etc. These system-defined events are already numerous, and will be expanded further in future releases as the needs become clear.
So how about classes? Well, here there is already a difference between the original released version from February and the "May Update" (also known as "platform update 1"). Both versions have the concept of extension methods, which pretty much mimic the C# behavior of extension methods. With the May Update a new feature was introduced that allows a more broader concept of extending where the extending class can actually define new instance member variables it can use. Note that in both cases the extension cannot access private or protected member variables - it adheres to scoping rules, and makes sense when thought of in the API-sense.

Let's leave this wall of text for what it is, and start looking at actual examples of extensions in the upcoming blog posts.

Wednesday, September 28, 2016

The Package Split

After reviewing the new paradigms in AX 7, it's time to put these paradigms to practice and review the current and future state of the Dynamics AX application code. To recap, a package corresponds to its own mini-modelstore, with its own layers and models. Each package roughly translates to a .NET assembly, meaning it's a unit of compilation. It also means one cannot over-layer from one package to another. Packages can "reference" each other, exactly like other .NET assemblies can reference each other. This is needed so one package can use features of another. This concept is being put to use in the standard application code, and the ongoing effort to accomplish this split-up is commonly referred to as the "package split".

Initially, the package split can be thought of in three main areas: the platform, the foundation, and the application. The platform is all the essential AX platform pieces that are not in the kernel but in X++, yet they are essential to the AX platform. Things like the users tables and the form to create and maintain those users. The foundation contains basic features that most of the application code will depend on. Things like the number sequence functionality. Finally, the application itself, which is all the business logic. Although you can think of those big three, the "application suite" as the package is called is slowly being split up further. You'll see packages like "general ledger" and "dimensions" that have been split off from the application suite. In a sense they sit in between the foundation and the application suite, because they are application logic but they are shared between different modules and functionalities.

So what is the use of having these separate packages? In a nutshell, it's all about maintainability and serviceability. As we discussed in the paradigm articles, having code in packages as opposed to a huge monolithic code base allows for faster compiles (since you're only compiling the code for your package / .NET assembly). Whereas AX 2012 and before would either require you to compile the whole application (or, in case you're into bad practices, take the risk of importing an XPO and just compiling forward or something), the new AX decreases the compile and deployment time needed by splitting up the code into packages (even though the compiler is already significantly faster than previous versions thanks to a rewrite). This has clear advantages during development (for example, being able to do continuous integration or gated check-ins, which is very difficult if your build takes 30 minutes or more. But it also has advantages for deployment, and testing. It's clear that a split in smaller packages means smaller and faster compiles and smaller and faster deployments. But considering that you cannot over-layer from one package to another and packages reference each other as .NET assemblies (DLLs), it means you can replace and update packages independently, as long as the API to its functionality doesn't change. Yes, I said API :-)

If you look at a dependency graph of the packages in the new AX, you'll find that the packages at the bottom are more like APIs. An API for number sequences, and API for dimensions, etc. At the top end, you'll have the ultimate consumer of those APIs. The application suite and other packages, providing the end-user functionality of the business logic. If a hotfix is needed on the number sequence functionality, the foundation package can be hotfixed and replaced without requiring a re-compile or re-deployment of the application suite - provided none of its APIs changed (which wouldn't happen for a hotfix of course). It also wouldn't require any code upgrade, assuming there are no over-layering models in the foundation package. And that is the gotcha…

Below a "simplified" view of the standard package hierarchy in AX7. Some packages were left out to show the idea. Since packages "reference" each other like normal .NET assemblies, the references are not transitive and usually the package at a "higher" layer needs to reference some of the lower layer packages as well. But clearly, everything references platform, some reference foundation, and it goes up from there.

So, to avoid over-layering and also keep one's code in a separate package to facilitate quick compiles and easy deployments of custom code, what is a X++ developer to do? That, my friends, is where extensions come into play. To be continued…