Live deployments of customizations are tedious, stressful and critical events for CRM Developers. Often done outside of hours to avoid disrupting the users (what about the CRM Dev’s, they have lives too!), if they go well it’s as it should be, if something goes wrong the lightbulbs are swapped to read and end users can be heard shouting SHOW STOPPER.
Despite the importance of production deployments not much time is spent thinking and improving the process, it’s often delegating to someone at the start of the project and they are left to get on with it. In the article below I look at the different aspects of the production deployment and highlight parts to improve.
Solution Overview
CRM developers need to know solutions inside out, upside down and back to front. Solultions organise and move customizations between CRM environments but be warned, they can cause problems during importing. When problems occur importing a solution, knowledge of solutions will help you diagnose and resolve the problem. It’s important you have a deep knowledge of the CRM SDK and why I am often saying Why CRM Developers should always start with the CRM SDK
CRM developers need a good knowledge of solutions because solutions are a tool you will use regularly, you need to understand
- What customizations are included in the solution file
- what customizations are excluded from the solution file
- working with multiple solutions
- Which customizations persist when mutiple solutions contain the same entity
- What the default solution is and how it works
- The difference between unmanaged and managed solutions
- When to use manage or unmanaged solutions
- dependant components
- common solution importing problems
CRM production deployments are often done out side of hours because importing a solution can freeze Microsoft Dynamics CRM but did you know you can import a solution asynchronisly using the CRM SDK – Asynchronous Solution Import
All CRM developers should have read CRM SDK Introduction to solutions
I have written a brief summary of the CRM SDK solution content in the article Understanding Solutions and how they work
Solutions use guids and sometimes solutions can get out of sync, this article explains how I got into and out of such a mess
Managed solution problems with out of sync solutions
Solution import Errors
Patching solutions
Solutions package CRM customizations, CRM 2016 has improved solutions to allow patching solutions, which allows you to import a smaller range of customizations instead of the whole entity and related customizations
read articles on patching solutions
Whether you are importing an older full solution or a patch solution, the solution is often only a small subset of the customizations being deployed.
Manual deployment steps
CRM deployments can include multiple manual steps, you should consider what manual steps need to be deployed in each deployment, list them out and practise them
- Unregistering plugins
- Workflows to deactivate
- Data to import
- Security roles to configure
- Configuration settings to update
- Workflows to re-point (workflows using data in the CRM instance, which changes guids between organisations)
- App pool settings to change
- update WCF dll‘s and config files
- update GAC dll‘s (if you have deployed any)
Unregisteringplugins is a manual step because solutions are additive, which is a fancy way for saying they don’t remove or delete customizations from a target CRM instance. Someplugins will need to be deactivated manually, usually using the Plugin registration tool.
Workflow re-pointing happens when a CRM records are used in the logic of the workflow. CRM records will have an inidividualguid, which is assigned to it on creation of the record. A record could have the same name in different CRM environments but will have different guids e.g. an Account named Hosk in DEV and PREPRODUCTION will have the same name but different guids. Workflows use lookup (EntityReference) which has the name of the entity and guidto uniquelyidentify the record.
Workflows which use CRM records will become unpublished when imported into a new CRM environment because the record guid can’t be found because it doesn’t exist. The manual workaround for this process is to open the workflow and lookup the correct CRM record and this is what I call Workflow re-pointing.
Solution Configuration
The structure of a solution can minimize the number of customization’s included in a solution and the potential problems when importing the solution.
A solution with all customization’s could grow when customization’s are added, taking longer and longer to import and maintain.
You can split up the solution to reduce the customization’s changed and deployed.
Two common methods of organizing the Solution file are
Release
You can organise solutions to releases/phases/sprints, including the changes for each major release of functionality. During development it’s easy for developers to work with the solution. Developers with experience of the project will find release/sprint solutions easy to understand but new developers won’t find the solution content obvious.
The release solution can lead to many solution files because solutions are not reused but it can stop solution files becoming to large.
- Release 2.1.1.1
- Release 2.2.1.4
- Release 3.1.1.0
Customization’s by type
I have seen a few projects organise solution files by type of customization’s
- Entities
- Entities and fields
- Lists
- Forms
- Code
- Javascript
- Plugins
- Workflows
- Custom workflows
- Reports
Its easy to create visual studio projects containing similar types of customization’s and it’s straightforward for developers touse.
Over time solution files can grow to be big, this can cause problems with several developers changing the same customization, an example is discussed
How do multiple developers work on a web resource within the CRM environment
There is no right or wrong way to structure a CRM solution, it depends on the project, number of customization’s, number of developers, etc. You need to create a solution organisation method which is logical and easily understood to the developers working on the project.
Document the deployment process
We create CRM development best practices and standards to ensure CRM development is created in standard way and to the required standard. All the code/customizations should be similar despite being created by different developers.
The deployment process should be documented and processes created to ensure deployments are done sequently, in the same way by different CRM developers. The benefit of documenting the deployment is transferring the build knowledge out of an individuals head and into a document other developers can follow.
- The benefit of documenting the deployment process
- deployment knowledge is shared
- steps are not forgotten/missed
- manual processes are documented
A standard deployment document, to act as a checklists to tick off during the deployment.
Pilots have a checklist when landing. Pilots are skilled individuals often with thousands of hours experience but without a checklist even experienced professionals can forget a step when the pressure is on.
The Checklist Manifesto: How To Get Things Right
During the pressure of a deployment, it’s easy to miss a deployment step, documentation and checklists will help you avoid this. Checklists and documentation can help do the deployment in the correct sequence (which might be important).
Don’t leave it to one person
Deploying to a production environment is a dichotomy, it’s a group of simple (and usually boring) tasks but it’s important it goes without problems. Problems with deployments will affect end users and impact the customer/developer relationship
When one person is tasked with doing deployments they get the deployment process down to a fine art and becomes a slick deploying machine. The downside is you have created a deployment dependency and no one else knows how to deploy or is confident to do it. There is pressure on the deployer moving project and when they go on holiday.
Deploying is a boring task which should be shared out between developers, minimum you should have at least two CRM developers who are comfortable deploying customizations in the production environment.
Don’t use personal users accounts
It’s good practice have a general service account to deploy CRM. This allows anyone to log onto the production environment with a user account who has the required security roles and access to the server.
If the deploying account is a specific user, when someone else logs in to deploy the solution, their user might not have the required security privileges.
Importing the solution is linked to an individual user, the user won’t want to share their user password details. A big problem can occur if individual users are not in the office and you don’t use a generic service account, you can’t do a deployment!
Backups
Rule 1. BACK UP everything, no arguments, no excuses, just do it because if it goes wrong and you can’t roll it back the person deploying will be responsible.
A live CRM instance should have multiple backup strategies such as long term backup strategy, weekly and disaster recovery.
Production deployments needs a short term backup strategy, only used if when a problem is identified during deployment and you need to roll back the changes to leave the CRM and related components in the same state pre deployment.
Make sure you are able to roll back all data changes/config changes.
for on-premise it‘s not only the database(s) which needs to be backed up
- Configs
- WCF services
- Windows applications (often running as services)
What to backup? consider different components and data
- Databases (CRM, Config, External)
- files on the CRM Server
- Data on other services (Sharepoint etc)
Data and SQL stored procedures are kept on a separate database, this best practice. It seems like added complexity but it’s separates the solutions custom tables and stored procedures from the CRM databases, meaning it won’t stop working when a CRM update/upgrade is applied.
Practice restoring
If you ask CRM Developers if they know how to rollback the changes they will say yes, you restore the database blah blah blah.
If you ask the same CRM developers to do it, they will suddenly look worried. Most CRM developers haven’t rolled back changes.
The last place you want to learn how to restore a database or roll back solution changes is on a live production deployment.
Practice rolling back changes so if you need to do it, you can.
Data
Guids and data can be a headache, if you import the data on each environment the guids will have different guids. Data used in workflows will break when you move them between environments. Deploying workflows can be a long arduous job if you have to re point imported workflows referencing CRM records.
A better method is to ensure the guids of your data are the same in all CRM environments. You can do this by importing data and specifying the guids, this can be done 3 ways
Manual Method
Exporting the data out of your DEV CRM (ticking for reimport), turning the files into CSV files, this will have the guid each row. You can import these into different CRM instances
Create a CRM SDK Console app
You can create a console app to export data out of your DEV CRM into XML or CSV. Code the console app to import data specifying the guids
Use a Tool – CRM Configuration Data Mover
CRM tool wizard Lucas Alexander has created a CRM configuration Data mover to move the data and keep the guids. He has written a great blog post to step you through using it
Introducing the Alexander Development Dynamics CRM Configuration Data Mover
Release notes
Creating a release note is a pain, it’s a boring manual job but…. they are very useful.
- Customer knows what is being imported
- You have an audit of production deployments (data, list of customization’s)
- You can see what bugs have been fixed and when
If you put your bugs into TFS you should be able to create release notes by creating a view and exporting. Like all boring jobs, automate the process and where you can and share the task between developers (e.g. share the pain and the knowledge)
Solutions
Solutions have version numbers, use them. Don’t deploy customizations without changing the version number.
It’s frustrating and confusing when CRM developers update customization but don’t change version numbers. This works OK in dev environments but when you have 3 or 4 CRM instances hosted on customer servers it makes it impossible to know what has been deployed.
Solutions have a description, you can use this to add more information about what was released.
Automate
If your deployment has manual steps which need to be done every time or often then automate those steps.
You can use simple bat files to copy files to the correct location and restart IIS, CRM Services. install DLL’s into the GAC, etc.
The danger of repetitive and boring tasks is they can be easily missed. I worked on a CRM deployment which had 11 CRM servers. The build process involved installing DLL’s to the GAC, updating solutions, updating Windows services and often importing data. When doing this repetitive task you enter a zombie like state
clicking, copying, clicking, copying, restarting
![]()
The process was boring and took two hours to do a simple deployment. I created bat files to copy files to the servers. A bat file on each server, deployed DLL’s and restarted ISS, restarted CRM services.
The benefits were two fold
- all dll‘s were all deployed and all services restarted
- The deployment took half the time
- The deployment wasn’t as boring
I’m sure there were better ways to automate the build but to create the bat files was simple and made the deployments less of a pain for me.
If you can automate parts of the build, do it because automating will avoid mistakes, take less time and isn’t as boring for the person deploying.
Plan and Preparation
On one project one deployment was a whopper, twice as big as the average deployment.
The deployment would take 4 hours but the size of the deployment cause us to plan the deployment in more detail and we were more prepared. We deployed in 3 hours and it was the smoothest deployment we had done on the project.
Time spent on deployment preparation leads to smoother deployments, make sure you are given adequate time to plan and prepare for the deployment
Pair deployment
You rarely need two people to deploy customization to production but having two developers doing the deployment has advantages
- It’s quicker with two people
- reduces mistakes
- Less likely to miss steps
- Not as boring with a buddy
- Deployment knowledge is shared
Two people will ease the pressure and work on a single person deploying. It’s keeps both people focused during the deployment. If the deployment goes wrong during the deployment, two heads are better than one.
Summary
This post was meant to be a small post to highlight a few observations I made during a successful production deployment. The more I though about production deployments the more parts to a successful deployment process I found.
What I found unusual was the lack of time spent on improving production deployments and ensuring they are efficient.
Automating as much of a deployment as possible should be priority because deploying customization’s takes lot of time and there are usually at least four CRM environments to deploy to.
he was asked if he enjoy promoting films, McAvoy gave an interesting response
