Thursday, April 25, 2013

Dynamics AX 2012 Compile Times

As I'm sure most of you are aware, compile times in Dynamics AX 2012 are a concern with more functionality being added. Especially on our build environments, which are (non-optimized) virtual machines, we are looking at around 3 hours for AX 2012 RTM/FPK and around 5 hours for R2. There have been discussions on the official Microsoft Dynamics AX Forums about this very topic, and there seem to be huge differences in experiences of compile times. After a lot of discussion with other people on the forums, and consequent chats with Microsoft people that are "in the know", I think it's pretty clear which areas one needs to focus on to optimize compile times.

1) The AX compiler was originally built when there was no talk about multi-core. So, as a result, you've probably noticed a compile only uses one thread. With today's trend of more cores but at lower clock speeds, an "older" machine (CPU type) may possibly perform better than a new one, or a desktop machine may perform better than a high-end server.
2) The communication between AX and SQL is critical. The communication with the model store is critical (AOS gets the source code from the model store, compiles it, puts the binaries back in the model store).
3) The model store is in SQL so SQL has to perform optimally.

To this end, I set out to build one of our customer's code bases (AX 2012 RTM CU3, no feature pack) on an "experimental" build machine. This code base has been taking an average compile time of 3 to 3.2 hours every time on our virtual AOS connected with physical SQL.


The new setup? A Dell Latitude E6520 laptop:

* Core i7-2760QM CPU @ 2.4GHz, 4 Cores, 8 Logical Processors
* 8 GB memory
* High performance SSD (Samsung 840 Pro), 256GB
* Windows Server 2012, SQL 2012, AX 2012 RTM CU4

Besides this hardware (2.4ghz clock speed - number of cores doesn't matter, SSD to maximize SQL throughput), the key elements of our setup are putting the AOS and the SQL server both on this same machine, and disabling TCP/IP in the SQL server protocols so that it uses shared memory instead. This is the least overhead you can possibly get between the AOS and SQL.

The difference in compile time is staggering. I actually ran it multiple times because I thought I had done something wrong. However, since this is an automated build using TFS, I know the steps and code and everything else is EXACTLY the same by definition. So.... drumroll!
(Note I didn't list some extra steps being done in the automated build explicitly so that's why it may not seem to add up...)


Old Build Server

New Build Server
Remove old models
00:00:27

00:00:03
Start AOS
00:01:26

00:00:25
Synchronize (remove old artifacts from DB)
00:06:52

00:05:57
Import XPOs from TFS
00:13:17

00:03:55
Import VS Projects
00:00:29

00:00:11
Import Labels
00:00:22

00:00:08
Synchronize (with new data model)
00:05:42

00:01:55
X++ Compile
02:29:36

00:41:28
CIL Generation
00:13:41

00:05:29
Stop AOS
00:00:10

00:00:03
Export Built Model
00:00:42

00:00:12
Total Build Time
03:14:43

01:00:59

So yes, the compile time got down to 41 minutes! We've actually switched using this machine somewhat permanently for a few customers, we'll be switching more. Now I need another machine for R2 compiles :-) I will post the compile times for R2 when I get to those.

Happy optimizing! :-)

16 comments:

  1. This is great stuff, Joris! Thanks for sharing your results. :-)

    ReplyDelete
  2. Excellent suggestion - just trying it now. Been looking for a SSD solution in order to minimize the I/O times but didn't looked at disabling the TCP/IP protocol to force a Shared memory selection.

    ReplyDelete
    Replies
    1. Just completed the test. The compile went from 4h to 3.5h with only Shared memory enabled. I think a key component is also the SSD disk ...

      Delete
    2. Thanks for testing! That setting was a theoretical gamble, we hadn't actually tested the difference between the two (yet).
      I know I have more testing to do to see what certain things have bigger impact over others.

      Delete
    3. In Microsoft Dynamics 2012 R2 CU7 a new command has been made available. AXbuild.exe. With that command you are able to build it much faster(around 10-25 minutes in total). As it runs parallel builds and is running as 64 bit. For more info see here http://msdn.microsoft.com/library/d6da631b-6a9d-42c0-9ffe-26c5bfb488e3.aspx

      Delete
  3. Thanks for your answer :) I would notice that I have run compilation UNDER Hyper-V machine :(( and unfortunatelly I did not plan to changemy HDD to SSD :( Anyway - how can I disabling TCP/IP in the SQL server protocols (I mean what program shall I use to do that ? SQL Manager ? some of AX configuration tools...?

    And from other hand - what are consequences of break of compilation process? Will I be able to use AX ? In fact I have run compilation becouse I could not start EP and Role Center...

    With thanks again :) and regards

    Witold

    ReplyDelete
    Replies
    1. It's in the "Sql Server Configuration Manager" under "SQL Server Network Configuration" -> Protocols.
      Of course that only works if the AOS and SQL are on the same machine. Putting SQL and AOS on the same box is the first major performance improvement for the compile anyway.

      Delete
  4. The Microsoft compiler team tested the shared memory setting as well and says that SQL is smart enough to use shared memory over tcp/ip automatically. So it sounds like you don't have to turn off tcp/ip necessarily, just turn on shared memory.

    ReplyDelete
  5. You can see the connection type with this sql statement
    select net_transport from sys.dm_exec_connections
    where session_id = "aos session id"

    ReplyDelete
  6. We have been doing the same for about a year. Virtual machines SUCK for doing compiles. A $1000 - $1500 development workstation box with SSD is the trick. While VMs are "convenient" they really don't have a place for real development. The time that is wasted easily pays for the dedicated machines very quickly.

    ReplyDelete
  7. One of our clients tested this out and felt the key piece was having the AOS and SQL on the same box. They did not use an SSD. The compile went from 3:45 to 1:38.

    ReplyDelete
  8. Thnx for sharing, very appreciated :-)

    ReplyDelete
  9. We use IMDisk to create a RAM drive (and store the SQL database there) on the compile machine - It reduces the compile time from just over 50 minutes (Hybrid HDD) to just a little over 30 minutes.

    ReplyDelete
    Replies
    1. Hi, Sashi, if you use SQL Server Enterprise Edition or Developer Edition, you can save on the size of the IMDisk Ram drive (and probably reduce compilation time even further), by applying database compression to the model store tables and indexes. Operational performance should improve as well.

      Delete
  10. Update.. Just got a new Dell Dimension T3600 development box here. Using a DUAL SSD Raid 1 drives, 64GB of memory and a PERC H700 RAID Controller which has caching and a built in battery..

    Compile time for RTM is down to 37 minutes.. The first pass of the compile, which is the metadata pass, takes 20 minutes.. So, if the compiler team can multi-thread that piece, I might be looking at less than 30 minutes to compile.

    ReplyDelete