1) The AX compiler was originally built when there was no talk about multi-core. So, as a result, you've probably noticed a compile only uses one thread. With today's trend of more cores but at lower clock speeds, an "older" machine (CPU type) may possibly perform better than a new one, or a desktop machine may perform better than a high-end server.
2) The communication between AX and SQL is critical. The communication with the model store is critical (AOS gets the source code from the model store, compiles it, puts the binaries back in the model store).
3) The model store is in SQL so SQL has to perform optimally.
To this end, I set out to build one of our customer's code bases (AX 2012 RTM CU3, no feature pack) on an "experimental" build machine. This code base has been taking an average compile time of 3 to 3.2 hours every time on our virtual AOS connected with physical SQL.
The new setup? A Dell Latitude E6520 laptop:
* Core i7-2760QM CPU @ 2.4GHz, 4 Cores, 8 Logical Processors
* 8 GB memory
* High performance SSD (Samsung 840 Pro), 256GB
* Windows Server 2012, SQL 2012, AX 2012 RTM CU4
Besides this hardware (2.4ghz clock speed - number of cores doesn't matter, SSD to maximize SQL throughput), the key elements of our setup are putting the AOS and the SQL server both on this same machine, and disabling TCP/IP in the SQL server protocols so that it uses shared memory instead. This is the least overhead you can possibly get between the AOS and SQL.
The difference in compile time is staggering. I actually ran it multiple times because I thought I had done something wrong. However, since this is an automated build using TFS, I know the steps and code and everything else is EXACTLY the same by definition. So.... drumroll!
(Note I didn't list some extra steps being done in the automated build explicitly so that's why it may not seem to add up...)
Old Build Server | New Build Server | |
Remove old models | 00:00:27 | 00:00:03 |
Start AOS | 00:01:26 | 00:00:25 |
Synchronize (remove old artifacts from DB) | 00:06:52 | 00:05:57 |
Import XPOs from TFS | 00:13:17 | 00:03:55 |
Import VS Projects | 00:00:29 | 00:00:11 |
Import Labels | 00:00:22 | 00:00:08 |
Synchronize (with new data model) | 00:05:42 | 00:01:55 |
X++ Compile | 02:29:36 | 00:41:28 |
CIL Generation | 00:13:41 | 00:05:29 |
Stop AOS | 00:00:10 | 00:00:03 |
Export Built Model | 00:00:42 | 00:00:12 |
Total Build Time | 03:14:43 | 01:00:59 |
So yes, the compile time got down to 41 minutes! We've actually switched using this machine somewhat permanently for a few customers, we'll be switching more. Now I need another machine for R2 compiles :-) I will post the compile times for R2 when I get to those.
Happy optimizing! :-)
This is great stuff, Joris! Thanks for sharing your results. :-)
ReplyDeleteAwesome stuff...thanks!
ReplyDeleteExcellent suggestion - just trying it now. Been looking for a SSD solution in order to minimize the I/O times but didn't looked at disabling the TCP/IP protocol to force a Shared memory selection.
ReplyDeleteJust completed the test. The compile went from 4h to 3.5h with only Shared memory enabled. I think a key component is also the SSD disk ...
DeleteThanks for testing! That setting was a theoretical gamble, we hadn't actually tested the difference between the two (yet).
DeleteI know I have more testing to do to see what certain things have bigger impact over others.
In Microsoft Dynamics 2012 R2 CU7 a new command has been made available. AXbuild.exe. With that command you are able to build it much faster(around 10-25 minutes in total). As it runs parallel builds and is running as 64 bit. For more info see here http://msdn.microsoft.com/library/d6da631b-6a9d-42c0-9ffe-26c5bfb488e3.aspx
DeleteThanks for your answer :) I would notice that I have run compilation UNDER Hyper-V machine :(( and unfortunatelly I did not plan to changemy HDD to SSD :( Anyway - how can I disabling TCP/IP in the SQL server protocols (I mean what program shall I use to do that ? SQL Manager ? some of AX configuration tools...?
ReplyDeleteAnd from other hand - what are consequences of break of compilation process? Will I be able to use AX ? In fact I have run compilation becouse I could not start EP and Role Center...
With thanks again :) and regards
Witold
It's in the "Sql Server Configuration Manager" under "SQL Server Network Configuration" -> Protocols.
DeleteOf course that only works if the AOS and SQL are on the same machine. Putting SQL and AOS on the same box is the first major performance improvement for the compile anyway.
The Microsoft compiler team tested the shared memory setting as well and says that SQL is smart enough to use shared memory over tcp/ip automatically. So it sounds like you don't have to turn off tcp/ip necessarily, just turn on shared memory.
ReplyDeleteYou can see the connection type with this sql statement
ReplyDeleteselect net_transport from sys.dm_exec_connections
where session_id = "aos session id"
We have been doing the same for about a year. Virtual machines SUCK for doing compiles. A $1000 - $1500 development workstation box with SSD is the trick. While VMs are "convenient" they really don't have a place for real development. The time that is wasted easily pays for the dedicated machines very quickly.
ReplyDeleteOne of our clients tested this out and felt the key piece was having the AOS and SQL on the same box. They did not use an SSD. The compile went from 3:45 to 1:38.
ReplyDeleteThnx for sharing, very appreciated :-)
ReplyDeleteWe use IMDisk to create a RAM drive (and store the SQL database there) on the compile machine - It reduces the compile time from just over 50 minutes (Hybrid HDD) to just a little over 30 minutes.
ReplyDeleteHi, Sashi, if you use SQL Server Enterprise Edition or Developer Edition, you can save on the size of the IMDisk Ram drive (and probably reduce compilation time even further), by applying database compression to the model store tables and indexes. Operational performance should improve as well.
DeleteUpdate.. Just got a new Dell Dimension T3600 development box here. Using a DUAL SSD Raid 1 drives, 64GB of memory and a PERC H700 RAID Controller which has caching and a built in battery..
ReplyDeleteCompile time for RTM is down to 37 minutes.. The first pass of the compile, which is the metadata pass, takes 20 minutes.. So, if the compiler team can multi-thread that piece, I might be looking at less than 30 minutes to compile.
Good stuff! :)
ReplyDeleteHave you considered using a RAMDisk for the build?
ReplyDeletei.e. move all IO operations into memory during the built itself; then once you've stopped the AOS on completion have a task to copy the exported model/modelstore from the RAMDisk to your SSD so it's persisted. Would be interesting to see the impact of that on compile/build times...