Wednesday, May 4, 2016

Design-Compile-Run Part3: Design, Compile, Run in AX7

This is Part 3 in my design, compile, run series. Please first read Part 1 about paradigms, and especially Part 2 on design, compile, run in AX 2012.

In part 2 I explained the design time, compile time and run time differences in AX2012. However subtle and vague they may be, they are there. We can (mostly) choose to ignore them, if we wish, but ultimately that is how the system works. In the new Dynamics AX (AX7), these three distinct phases are hard facts that you cannot work around. There is a design time in Visual Studio where you create your code and your metadata. There's a compile time where that code and metadata is taken and compiled into DLLs. And then there is a run time that can run those DLLs. And although the compile time is integrated into Visual Studio (the design time), it's essentially a command-line utility that Visual Studio calls to invoke that compile time. Let's look at what this looks like.

I'll re-use the graphics from the previous post, in reverse order, to build back up to the paradigms in the next post.
First, let's look at the tools we use in AX 7. As you probably know by now, AX 7 no longer has a client. Everything runs server-side, and there is some browser code (arguably that could be considered a type of client) that displays everything from the server. So the AX 2012 client which hosted both the design time, compile time and run time is entirely gone, so there is no tool left that spans all three phases.


Immediately noticable is that each tool handles a specific phase, there is no overlap anywhere. There's design-time tools (Visual Studio), compile-time tools (xpp compiler - xppc) and the .NET run-time running with the AOS kernel in IIS. Note also that there is no X++ interpreter anymore. X++ is now a 100% managed language running in the .NET CLR. The compiler is now rewritten entirely and compiles directly to .NET assemblies (and is scalable... give it more memory and more CPU and it will go faster). Of course, this has some implications to the traditional way I described in the previous post. For one, you always have to compile everything (although there's good news here, we'll describe what "everything" really means in the next post). If you have a compile error, you don't have anything to run. This also means you actually have compiled binaries to deploy. The run-time can't do anything with code, it can only run binaries. Which means for deployment purposes, you're deploying binaries not source code. So no merging and all that because that is a design-time task.

Next, let's look at the next slide of what "file formats" exist in each phase.


So, yes, XPOs are finally gone! And make no mistake, the XML files are not just a replacement - they ARE your code files now. Since design-time is separated from run-time, you don't "import" your code into a server. The code _is_ a bunch of (XML) files on disk. The next question that comes up a lot is then - but how do I run the code in Visual Studio? Well, just like you would be creating an ASP.NET app or such, Visual Studio will run the compiler for you, then copy the binaries into a local IIS server, and start the IIS process. Visual Studio doesn't *run* your code, after compiling it deploys the binaries and then opens a browser - exactly like a C# ASP.NET project. That means when you edit code and or change forms from within Visual Studio, you are actually editing XML files. This has a few important implications!

1) You _cannot_ "share" a development box for several reasons. If two developers are editing the same file at the same time, the last one that saves wins and overwrites. Think of editing a .TXT file in Notepad. If two people open the same file at the same time, the first one makes a change and saves. The second (who doesn't have the first one's changes) makes a different change and saves. Essentially, only the second person's changes end up in the file. The other reason is similar to AX 2012 - if someone has work in progress you could have compile errors, which now in AX7 entirely prevent you from generating the assembly DLLs for the runtime.
2) Since all code is files, and we are working inside Visual Studio, source control is a breeze! Where in previous releases of AX there was always an export to XPO and then check into source control (essentially a two-step process in different formats), you are now editing EXACTLY what you will check into source control. More on this in the next post when we talk about the new paradigms.
3) Some "automatic" relationships that existed when the AOT was part of the runtime (let's say, renaming a field) no longer resolve. If you rename a field, you'll have to get the list of references (or do a compile to get the errors) and fix the artifacts and their files (for example, form controls etc.). This is a good thing: in AX 2012 this could cause issues when using source control - a field rename would fix some things automatically in your AOT, but wouldn't check-out and re-export the XPOs of those "fixed" artifacts, probably causing issues for other developers, builds, etc.

Of course, these are the file formats for the artifacts that the phases use. But now we have another new concept - file formats for DEPLOYING those artifacts.


The concept of an AxModel still exists, and it's now essentially just a zip file with all your model's XML files in it (plus the model manifest, now called the "descriptor file"). AxModels are a format to distribute code around, but mostly between different development parties (such as an ISV, a Partner and a Customer). To keep source code in sync between developer's machines at the same company, we use what we everyone else is using: source control. Although some new LCS features require you to use VSTS (Visual Studio Team Services) with TFVC (Team Foundation Version Control), for source control purposes only you can use whatever Visual Studio or its plugins support. You can use VSTS with git, you could use GitHub, etc. Note that VSTS with TFVC is currently required for project management and code upgrade features - support for other source control systems is being evaluated and may get added in later versions.
The concept of the model store needs some rethinking, we'll discuss that in the next post. But as we discussed here, the run-time just needs the DLLs of your custom code. We don't want you to just copy DLL files around, so there's a concept of a "deployable package" which is essentially a collection of one or more DLL files with scripts to deploy it. Today there are some manual steps to deploy that deployable package, but automation for this is just around the corner. So, when deploying code to a test environment or production environment, the deployable package is the vehicle to distribute your compiled assemblies.

Both the model file and the deployable package can be outputs of your automated builds. We'll discuss builds and automation some other time.


Next post, let's put these tools and file formats in perspective and review the paradigms and architecture in AX 7.

Tuesday, May 3, 2016

Design-Compile-Run Part2: Design, Compile, Run in AX 2012

In Part 2 of this design, compile, run series we'll look what design, compile, run actually means in AX 2012.
If you haven't read Part 1 about the paradigms in AX 2012, please do.


We've looked at layers and models in Part 1 of this blog post series. In versions prior to AX 2012, these layers were stored in a physical file (AOD file) - each layer had one file on disk. In AX 2012, this changed into a Model Store database - containing all the layer and model's code, metadata as well as all compiled p-code and generated CIL binaries. Let's look at what these concepts really mean and represent.

First consider the metadata and code. When we say metadata we are talking about the form definitions, table details like relationships, indexes, schema representation etc. They are not "code" per se but represent an artifact in a meaningful way that is needed during development and parts of running the application, and some of it is used by the compiler as well.
The second thing the model store contains is the p-code and generated CIL. This is the binary form of the compiled code and metadata that is used to actually run the application. X++ originally (in AX2012 and before) is an interpreted language, and the p-code represents an easy way to run the code. The advantage of this is that parts of the code can be adjusted and only those parts have to be recompiled for the runtime to work. The disadvantage is that errors can easily occur at runtime when individual parts are changed without recompiling everything else in the system that may have a dependency on it. Additionally, since it's p-code that is executed by an interpreter at runtime, the performance of the execution can't match that of a properly compiled binary.
The introduction of CIL in AX2012 changed this story slightly. After the p-code is compiled, an extra process needs to be run that will translate the p-code into .NET assemblies. This step may fail if, as explained before, the application has errors due to individual parts having been changed without recompiling and resolving all consequential errors. The advantage is of course that X++ code can be run inside the .NET CLR. The disadvantage is it conflicts with the traditional change&save model of X++ since this can cause issues. Additionally, if CIL generation runs into errors the application is potentially in an unknown state. It is entirely possible to have the CIL assemblies be in an "old" state (since new compile failed) while the X++ p-code has the latest changes. Essentially that means you could be executing different versions of the code whether you are running X++ interpreted or in the CLR if your development and deployment processes aren't solid.

Since traditional development for Dynamics AX occurs inside the AX client, the underlying structure and process is not very visible, which could lead to broken deployment processes. It's important to understand these details to have a solid AX 2012 development and deployment process. It will also make the AX7 processes clearer. Based on the above information, let's make the following three buckets: design-time, compile-time and run-time. Within the model store, there are both design-time artifacts and run-time artifacts. Note that the run-time does NOT have models or layers.


Why? Well, at the time of compilation the compiler will take the top-most version of each object artifact (think "granularity" as explained in part 1) and compile that into binary p-code (which then is used to generate CIL). That makes sense: first, you can only execute one version of the code so why have binary versions of each layer or model? Second, one piece of code may reference another piece of code - and each may exist in different layers, so the compiler needs to make a decision on what code to compile against. So all in all, the compiler will "flatten" all the code and metadata into one version of the truth, and convert that into binaries.

Now that we understand the underlying distinction between design, compile and run-time and what artifacts each contains, let's review what tools are used for each.


I realize this is a simplification, there are other runtimes like BC.net and SharePoint Enterprise Portal, but in general times this a good representation for the sake of this discussion.
- The AX32 client is the most comprehensive. It contains design-time tools, compile tools and an X++ interpreter to run the p-code. Note that it doesn't have the .NET CLR so it cannot run the X++ CIL.
- The AX32Server AOS is mostly a run-time which can run both the X++ interpreter for p-code as well as the .NET CLR to run X++ CIL. Note that in theory this has a server-side compiler in it that is used by the AxBuild process but one would never explicitly use it.
- The AxBuild process was introduced in AX 2012 R2 CU7 and is available ever since. It's strictly to compile, and although it technically starts the AX32Serv executable to compile, we'll consider this the compiler.
- I also included Visual Studio as a design-time tool, although from an X++ point of view there isn't much functionality and its mostly used for reports or non-X++ code (C# etc).

Finally, although all of this is stored in the model store, there are different file representations of different artifacts. These are file formats and vehicles to transport these artifacts (i.e. deployment artifacts).


First is the traditional XPO file. It's essentially the closest to a "code file" you can get with X++, it represents an object and its metadata and source code. Although one could have an XPO containing all customizations to an environment, it's no longer recommended as a vehicle of deployment but treated for what it should be: a code file. Meaning it's good to share bits and pieces of code among developers, as well as source controlling code artifacts.
Next is the AxModel file. This is essentially a collection of source code, typically representing a full solution of a responsible party. For example an AX customer's custom code, an ISV's product, a VAR's add-on, etc. Note from the diagram you see the AxModel file, although a binary format (actually it's a .NET assembly), is still just code and metadata. This means just like the XPO, installing it on an environment only gives you the code and you should still compile everything.
Finally the model store, which contains all code, metadata, p-code and CIL can be exported as an AxModelStore file, or a SQL backup can be taken to move it. Since this contains everything, there is no need to recompile anything when moved. The model store file or model store database backup is the closest we get in Dynamics AX to moving a compiled binary.
As I keep reminding people I meet that have questions around all of this, I like the analogy of a program or windows update. When there is an update, does the software vendor send you some C++ files (i.e. XPO)? Or is it more a Linux-like source code package that can be compiled (i.e. axmodel)? Or do they send you a new version of the DLL (i.e. model store)?


Next part, we'll transition into AX7 based on what we reviewed in this article.

Sunday, May 1, 2016

Design-Compile-Run Part1: 2012 Paradigms

If you haven't already, please read the intro first.

In this first post, we'll discuss development paradigms such as modelstore, layers, models, AOT artifacts and more. If you're reading this chances are you know most of these details, but I encourage you to read through this as the details are important to understand the changes being made in the new Dynamics AX (AX7).

To roll into this series, we'll discuss some typical paradigms. Dynamics AX has a system of models and layers (layers have always been there, models were introduced in AX 2012). The layers allow responsible parties to entirely separate their original source code and design artifacts, while allowing the next party (in the chain of vendor to customer) to customize all the code delivered by the previous parties. The number of layers is fixed (although the amount of layers and their names has changed over different versions of AX), and are named after the typical parties involved in an implementation. Within each layer, any number of models can be created. There is always a default model in each layer (called "XYZ model" with XYZ being the name of the layer). Let's look at what these layers and models represent and how they work.

Assume the case of a table delivered by an ISV, in the ISV layer. It contains one key field, a method, and it overrides the validateWrite() system method. The partner implementing this solution has made some more industry-specific changes, adding a field and extra validations in the validateWrite() method. To do so, they over-layered the table in the VAR layer. Additionally, the customer decides to change the labels on the ISV and the VAR field, as well as add a method.


Essentially, the table itself is now created in ISV, but different parts of it exist in different layers. Now, imagine the customer hiring another third party to make some more changes, who implement those changes in their own model in the VAR layer. Now the table also exists in multiple models in the VAR layer.


Looking at the customizations that were made, there are two distinctions to make. Some elements of the table are CHANGED in a particular layer, other parts are completely NEW. As we'll see in later posts, in the new Dynamics AX (AX7) the terminology for this is a customization (change) versus an extension (new). When a change is made to an existing element, that element is 'copied' from the lower layer (say, ISV) into the layer it is being customized (say, VAR). At this point, the element exists in both layers, but the compiler and runtime of AX will always only use the higher layer version of the element. The name 'element' is quite ambiguous here. In our table example, fields and methods are definitely elements that can individually be added or changed (over-layered) in different layers or models. However, there are certain sub-elements that CANNOT be individually over-layered. For example on tables an index is an element that can be individually over-layered, but individual fields in the index are not. The index as a whole will always be copied into the higher layer when customizing. Another example is the datasources node on a form. As soon as you add a method to a datasource or add a new datasource, the whole datasources node is copied into your layer including all the other datasources, methods, fields, etc. Other elements, like Menus for example, do not allow any sub-elements to be over-layered and are always entirely copied. The concept of what element level objects can be over-layered is often referred to as the granularity of objects. Some objects are entirely granular (for example classes, since both the class itself, its methods, and event subscriptions can all exist in different layers and models) and some objects aren't granular beyond the parent level (like menus). This of course has a major impact on upgrades of code. If the lower layer is updated, the higher layer needs to compare any over-layered elements. So obviously, the less granular an object is, the more work an upgrade will take.
Another effect of the granularity has to do with models within a layer. Models have the limitation that within one layer, an element can only exist in 1 model. So the more granular an object's elements are, the more it can be customized in different models within one layer. In our diagram's example, if tables were only granular to the table level itself, the VAR layer couldn't have had a new model that adds another method. With the granularity of tables as they are, a method or field can be added in the same layer in a different model. However, the new VAR model cannot also add a customization to validateWrite(), since that method in the VAR layer already exists in the other VAR model...

Tip: to see granularity of objects, go to your user options in Tool > Options and on the Development tab under "Application object tree" set layer to "show all layers" and model to "show on all elements". Once those options are on, you will see the layer and model listed for each element in the AOT. If an AOT does not show a layer/model, it means it's not granular and you have to look at its parent. For example on forms, note how all the controls are granular (they list a layer and model), but individual datasources do not, only the datasources node as a whole does.



In part2, we'll review the "design, compile, run" principles in AX 2012.

Thursday, April 28, 2016

Design-Compile-Run Intro

Over the last few years I've talked a lot (both online and offline) about how we should all strive to put software engineering practices back into AX. What I mean by that is that we've become lazy in how we do and manage our development, because AX has certain features that allows us to do just that. But with the product growing bigger, the features becoming more advanced and the projects getting bigger and more complex, the need to go back to those good practices has grown as well. As a result, the demand for Version Control and other development best practices for AX has become bigger. With the release of "the new" Microsoft Dynamics AX, some major shifts in this regard are happening. All for the better, but for some people this may represent a departure of some AX-specific practices.

Starting next week, I will be posting this blog post series "Design - Compile - Run" as a guide into paradigms and their meanings, starting with AX 2012 and leading into the new Microsoft Dynamics AX. I hope this will help serve as a guide to get your AX 2012 development processes in line and ready for the new AX. Discussing some details of designing, compiling and running AX is valuable information and will help you understand the new AX better.


Part 1: AX 2012 Paradigms
Part 2: Design, Compile, Run in AX 2012
---
Part 3: Design, Compile, Run in the new AX
Part 4: the new AX Paradigms

Monday, March 14, 2016

(The New) Dynamics AX - Content

If you haven't heard, the new version of Dynamics AX is out and available. Lots has changed and there's plenty to learn. Beyond the blogosphere there's some official places to find out all about the new features. We held our technical conference about a month ago, and for partners and customers that content is now being made available online.

So here's some links you want to bookmark:

Technical Conference 2016 Session Recordings (partners)
Technical Conference 2016 Content (customers & partners)
Dynamics AX Wiki (Official help site, public) - expect lots of updates here (what did you expect, it's a wiki!)

Friday, September 18, 2015

General Update

In case you hadn't noticed... I haven't posted in a while. For those of you who are connected to me in various ways, you have probably also noticed I had a job change and now work for Microsoft Dynamics AX R&D as a senior solution architect.

Essentially, in my new job I will be able to continue blogging and speaking at conferences. In fact, I will be at the AXUG Summit in October and speaking there. This blog will remain my personal blog and the open source projects will be maintained as well (more on that coming soon). Of course as a personal blog nothing I say or post here is backed up by Microsoft (see disclaimer at the bottom of the pages).

Currently we are hard at work on AX7 and I can't wait to start sharing and blogging about all the goodies for developers. But until then, it will probably be a bit quiet here (except for some planned improvements on the AX2012 build automation). Please connect with me on Twitter and stay in touch.

Tuesday, November 11, 2014

AX2012 Extension Framework

In a lot of ways, us X++ developers have gotten lazy over the years. If you take a step back, there are a lot of things we do today that you probably would never do in any other programming environment. Some of it is because it's just quick and easy, regardless of the less quick option being more robust or cleaner. In other cases, it's just because "that's how it's done" (tm) in the standard Microsoft Dynamics AX code.
Every now and then some new feature in the language or frameworks will come along, and it's either not documented or nobody really knows much about it. And in most cases it's barely used in standard code. Even things that everyone knows about, like events, are totally underused in the standard code base. To cut Microsoft some slack, they obviously have millions of lines of code that would have to be changed to adopt all these new features. And that obviously won't happen overnight.
Anyway, I wanted to point out this under-documented framework called the Extension Framework. I've only seen some minor blog posts about it, but I couldn't find it on MSDN or Technet anywhere. Regardless, I think this has great potential - especially for ISVs or Partner solutions that need to become more plug & play.

So, in this blog post we will create a totally unexciting set of classes. You'll recognize the standard AX pattern immediately. First, I have a base enum defining the different Dynamics products:


Next, I have a base class named Dynamics.


It has a new method that's made private (forcing people to use the static constructor) and a run method that has an exciting infolog message.

private void new()
{
}

public void Run()
{
    info("base class");
}

Next, we'll make a derived class for each of the Dynamics products:


Each one will extend the base class and override the Run method to show a different infolog. Below is the code for the DynamicsAX and DynamicsCRM class, you get the idea for the other classes...

public class DynamicsAX extends Dynamics
{
}

public void Run()
{
    super();

    info("Dynamics AX");
}

public class DynamicsCRM extends Dynamics
{
}

public void Run()
{
    super();

    info("Dynamics CRM");
}

Great. So now that we have our sub types, we can create the "classic" construct method on our base class. On the base class "Dynamics", create a new static construct method, which takes our base enum as a parameter, and returns the correct sub type for it.

public static Dynamics construct(DynamicsProduct _product)
{
    Dynamics dynamics = null;

    switch (_product)
    {
        case DynamicsProduct::AX:
            dynamics = new DynamicsAX();
            break;
        case DynamicsProduct::CRM:
            dynamics = new DynamicsCRM();
            break;
        case DynamicsProduct::GP:
            dynamics = new DynamicsGP();
            break;
        case DynamicsProduct::NAV:
            dynamics = new DynamicsNAV();
            break;
        case DynamicsProduct::SL:
            dynamics = new DynamicsSL();
            break;
        default:
            throw error("Unsupported product");
    }

    return dynamics;
}

Nothing too crazy here, a very traditional pattern you see a lot in Dynamics AX. Finally, we can create a static main method on the base class to run it. For the sake of this article, we'll just hard code the product to be AX (because that's what we're really here for, isn't?). Feel free to create menu items with parmenum or something, but to keep it short and to the point, hard coding will do. So again, on base class "Dynamics", add the main method:

public static void main(Args _args)
{
    Dynamics dynamics = Dynamics::construct(DynamicsProduct::AX);

    dynamics.Run();
}

And of course, when we run this, we'll see the infologs showing. The base class declaring it's the base, and the sub type declaring it's Dynamics AX. Perfect!

At this point, let's take a break and step back. What's wrong with this pattern? First of all, there is no way to extend this code base without either changing the code or overlayering (the construct method). Secondly, from a more design approach, it seems bad that the base class has to have explicit knowledge of all the sub classes. Put together, this is not very extension-friendly. If we would have two ISV products that both extend this, someone will have to merge the code. And nobody likes merging code.

So, extension framework! The idea is to use the new attributes feature in the X++ language to decorate the classes, and make a constructor that can dynamically find the right sub class, assuming there is one. So first, we need to create a new attribute. An attribute is basically a class that extends the base SysAttribute class. An attribute can have different properties, some mandatory, some optional. In this case, we want to use the attribute on a class to tie it back to one of the enum values. So, our attribute should have a property to hold that enum value. The base enum was called "DynamicsProduct" so we'll declare a member variable to hold that.
We'll call our attribute class "DynamicsProductAttribute".

class DynamicsProductAttribute extends SysAttribute
{
    DynamicsProduct product;
}

Since we don't really have class properties in X++, we need to create a set/get method, or as X++ likes to call it, a PARM method. Basically, a way to access the member variable to read or update it.

public DynamicsProduct parmDynamicsProduct(DynamicsProduct _product = product)
{
    product = _product;

    return product;
}

Now, to make this a mandatory property on the attribute we can add it as a method parameter on the new method. So, override the new method on the "DynamicsProductAttribute" class.

public void new(DynamicsProduct _dynamicsProduct)
{
    super();

    this.parmDynamicsProduct(_dynamicsProduct);
}

So, that's really it for the attribute class. Now we can actually start using it to decorate our sub classes. No need to touch the base class, but on all the sub classes, we want to add the attribute in the classdeclaration. Again, I'll just show the example for the DynamicsAX and Dynamics CRM classes:

[DynamicsProductAttribute(DynamicsProduct::AX)]
public class DynamicsAX extends Dynamics
{
}

[DynamicsProductAttribute(DynamicsProduct::CRM)]
public class DynamicsCRM extends Dynamics
{
}

Decorate each sub-class with the attribute and the correct enum value. This approach makes a whole lot more sense. Now each sub-class is declaratively tying back to a specific enum value. So finally, we of course need to change the constructor on the base class to make use of this at run time. This is really the only part where there's a "framework" really in place. On the base class "Dynamics", we're going to change the static construct method.
Now, we can ask the extension framework to grab us the class with a specific attribute. Of course we're interested in the DynamicsProductAttribute, but more importantly we're interested in the class that has that attribute but also has a specific value for its property (in our case, the enum DynamicsProduct). The extension framework just asks us what base class we want to grab sub classes for, as well as an INSTANCE of the attribute it should have. The reason we need an instance is because it needs all the properties (in our case we only have 1, but you could have several properties).
To get all this done, we can call the static "getClassFromSysAttribute" method on the "SysExtensionAppClassFactory" class. We give it the base class name ("Dynamics") and the attribute we want, which has to be an instance, so create an instance and set the correct property value. Your new and improved (and shortened) construct method should look like this now:

public static Dynamics construct(DynamicsProduct _product)
{
    Dynamics dynamics = SysExtensionAppClassFactory::getClassFromSysAttribute(classStr(Dynamics), new DynamicsProductAttribute(_product));

    return dynamics;
}

Since our static main was already using construct, you can just re-run the main and everything should work.
Note that this class factory business caches EVERYTHING. If you've run it once, and then re-run after making changes, it may not pick up new classes or changed attributes. From the Tools menu, you can run Tools / Caches / Refresh Elements to flush the cache so it will pick up your changes.

Let's review what we can do now.
- To extend this solution, we can add a new enum value. Create a new class, use the product attribute and tie it to the new enum value, and everything will work. No need to change any code, only adding code and decorating it!
- Of course, we use an enum here which isn't necessarily extension friendly, but there's nothing stopping you from using other data types, multiple properties, etc.
- The "choice" of which base class to use could be from a parameter per company, or based on a value in a record, whatever. The possibilities are bigger than you might think. Of course the traditional construct has this as well, but the ease of extending and swapping things out without having to mess with the code are pretty great.