Lone Wolves

I have been mulling a long time how I am going to keep track of SugarCRM, the custom modules I build and and changes to the core code that I need to make. The latter is sometimes unavoidable because some functionality cannot be built any other way. Also, sometimes you run into a bug that you need to fix, but you cannot wait for the next Sugar release. Of course, all of this needs to be done in a way that is easy for developers to work with. I think I have finally found a way that is workable and keeps everything under version control.

I will present my workflow in two articles. This article shows you how I keep SugarCRM itself under version control, how I deal with deployment and how I do upgrades in the face of changes to the core code. In “Build custom SugarCRM modules in Subversion” I will present how my custom modules—which are also under version control—fit into this system. Hat-tip to Leonid Mamchenkov for his work and insights.

0: Table of contents

• 1: Notes on this setup
• 2: Setting up a development server
• 3: Set up Subversion and prepare for importing Sugar
• 4: Import Sugar into Subversion
• 5: Import a new release
• 6: Update the trunk and working copies to a new release
  • 6.1: Update the database
  • 6.2: Update the source code
• 7: Deploying and updating live installations
• 8: Footnotes

1: Notes on this setup

The setup I am describing below allows me to keep SugarCRM under control when I make changes to the core code. As I explained above, sometimes you simply have to change the core code. Either to fix a bug that upstream does not fix fast enough or because a feature cannot be implemented in an upgrade safe way. There is however a fundamental problem with keeping any web application under version control: You can version control the source code, but not the database.

The way this dilemma is usually solved is by making the database follow the source code. The source is always the master and the database always the slave. When the source code and the database disagree then the source is leading and the database adapts. Ideally this happens automatically. With SugarCRM this is a problem however. Custom fields created in Studio for example are stored in the database, not in vardef files, but those custom fields are referenced in e.g. the viewdef files. Now you have a circular dependency between the source code and the database and as any programmer can tell you, circular dependencies are evil.

I break this circle by forcing the source code to be leading. Luckily SugarCRM has a very useful function called “Quick rebuild and repair” that can detect changes in the vardefs and make the database follow it. This does however mean that all such changes need to happen in the vardefs to begin with. This boild down to: “Don’t use Studio”. If you use the below setup while you manage custom fields, layouts and relationship with Studio then you will run into conflicts^*.

I manage all my custom fields, layouts and relationships with a custom package that I can install through the module loader. This package is also kept under Subversion control but sits in a separate repository. “Build custom SugarCRM modules in Subversion” explains how this works exactly. The base SugarCRM is what I keep under version control with the method shown below. The only updates made to it are my changes and fixes to the core code, and updated releases from SugarCRM upstream. Nothing else.

2: Setting up a development server

For my system I use a separate development server that has been configured in such a way that I can quickly make a subversion checkout and have a new copy Sugar running. No Apache configuration required because it’s all automated away.

Take a look at my article “Easily develop and deploy web applications from subversion” to see how I have set up Apache. What it boils down to is that I can make a Subversion checkout in a predefined directory, and Apache automatically picks up that subdirectory as a separate subdomain. So, if I do this:

Apache will see the “my-checkout” directory and (thanks to wildcard DNS) make it available on http://my-checkout.dev.example.org automatically. I did a similar trick for the MySQL database because I don’t want to manually grant database rights every time I create a new SugarCRM checkout. This is easy to do because MySQL supports wildcards in the database name when you grant right. So, manually I created a new database called sugar-vendor-510b (because I will be installing Sugar 5.1.0b today), created a new MySQL user called sugar and then I granted him rights on the database sugar-*. This way the new user will have rights to any database whose name starts with “sugar-”.

3: Set up Subversion and prepare for importing Sugar

Now it is time to import the first SugarCRM. I assume that you have an empty Subversion repository at http://svn.example.org/sugar to which you have rights. If you need help setting up Subversion under Apache, check out my older article “Apache and Subversion authentication with Microsoft Active Directory”. Start by creating a standard Subversion layout, but notice I do not create trunk/ yet.

Now go to your Apache checkout directory, make a checkout of the vendor branch, unzip the Sugar tarball and put the files in the checked out directory.

Before you can install Sugar you need to make sure that it can write to the directories. When you upgrade Sugar it wants to be able to write to all the files, so give it full access.

Now you can go to to http://sugar-510b.dev.example.org/install.php and install Sugar using the webinterface. Use the database and MySQL user that you created earlier. After installation we need to edit a few files before we can perform the actual import. SugarCRM stores the name of the database and the full URL of your sugar installation in config.php. We need to abstract these so that we can easily change these without having to edit config.php itself. I create two files name .hthost and .htdatabase that contain the hostname and database name respectively. Then I change the config.php script so that these variables are read from the file.

In config.php

You can even automate it further, but then you will need to change the config file manually when you deploy to a different server. Notice that the database name is now “sugar-510b”.

Next we need to add a few directories to the cache/ directory. The install script does not automatically create these but we do want to keep track of them (so we can ignore their contents later on). Please read my previous article The SugarCRM cache directory demystified to find out why you can ignore the contents of these directories.

The last thing to do is to rename the sugar-vendor-510b database. I do this so that I don’t accidentally write to the database anymore. I renamed it to vendor-sugar-510b so that it does not start with sugar- anymore and the sugar MySQL user does not have any rights to it.

4: Import Sugar into Subversion

Now you can import Sugar into Subversion. Start off by setting some ignore properties on the root directory. Note that to make it multiline you press Ctrl+Return, not just Return!

Next, add the cache directory, its subdirectories and set ignores on those as well.

Now we can add everything that is not ignored and commit it

The last thing to do is to copy this vendor branch to the trunk and copy the vendor-sugar-510b database to sugar-trunk.

Check out a working copy

Making a checkout is pretty easy in this setup. First I make a copy of the vendor database. I copied vendor-sugar-510b to sugar-test. Then I make a checkout using the following commands.

Now I can go to http://test.dev.example.org and log into my working copy of SugarCRM.

5: Import a new release

Importing a new release into Subversion is quite easy. First I copy the vendor branch to a new branch that will get the update.

I copy the vendor-sugar-510b database to sugar-vendor-510c, make a checkout of the new vendors/sugar-510c in my checkouts directory and then I install the update package from SugarCRM on that installation. After the update completes I commit vendors/sugar-510c, but do check the config.php file before you do so. SugarCRM has a tendency to regenerate its config file and when it does, you need to re-apply the changes that load the database and hostname from .htdatabase and .hthost.

Finally rename the database to vendor-sugar-510c to protect it against accidental writes. I now have a clean copy of both the 510b source code and database as well as the 510b and 510c database.

6: Update the trunk and working copies to a new release

Updating the trunk or any other working copy is slightly more work, because it is a two-step process. You don't know what kind of changes the update will make to your database so you need to actually install the update. You cannot simply update the source code and run “Quick rebuild and repair”. Sometimes SugarCRM makes very invasive changes to your database that are not reflected in the vardefs. For example, the update from Sugar 5.0.0a to 5.0.0b encrypted the SMTP password field in the database. But installing an update overwrites the source code files so you will loose any changes that you made to core files. The solution is to update the source code and database separately.

6.1: Update the database

If you want to update a working copy then you will need to commit your changes. For safety you may want o backup your database. Now make another checkout (or export) of that branch or trunk to a temporary directory. For example, make a checkout of the trunk to /var/checkouts/temp. Configure this checkout to user the sugar-trunk database (or whatever database you want to upgrade). Now go to this temporary SugarCRM and install the update in the regular way. When the update tells you that it has found local changes, tell it to overwrite all files with the new version. When the update is complete then your database has been upgraded. You can remove the temporary checkout directory. Note: Do not commit this temporary checkout because all your custom changes have been undone!

6.2: Update the source code

Updating the source code is done using a regular Subversion merge so that any modifications we have made will remain intact. Remember that we copied vendors/sugar-510b to vendors/sugar-510c? We can use this to generate the exact changeset between the two versions. Below you can see how I would update the source code of my checkout of the trunk to the new Sugar version.

Resolve any conflicts between the SugarCRM update and your own changes and commit.

Other people can now simply run “svn update” on their working copies of the trunk. Now you have updated both the database and the source code so you can use Sugar again. The only tricky part about this bit is that the database and the source code should be updated at the same time. If you are working with multiple developers who use their own working copies of the trunk but share a common sugar-trunk database then one developer needs to make this update. The other developers cannot use Sugar in the period in between the database update and the source code update.

7: Deploying and updating live installations

Deploying a live installation is simple. It is exactly the same as checking out a working copy. The only difference is that this time you probably need to configure Apache because most likely you will not be running the live copy out of your /var/checkouts directory.

Updating a live installation is also pretty much the same. First update the database and then update the source code. If there have been no new SugarCRM updates then there is no need to update the database. If there were then you need to check out an appropriate working copy from your repository and use that to update the database by installing the SugarCRM update. It works the same way as updating a working copy because a live deployment is in essence just another working copy.

After the database has been updated you can run “svn update“ followed by a “Quick rebuild and repair” to update the codebase and propagate the last changes into the database layout.

8: Footnotes

…conflicts. Such circular dependencies are one of the many ways in which SugarCRM is hostile towards maintenance under Subversion. Another one is its tendency to simply delete and recreate some cache directories during upgrades, after which you lose your .svn directories and Subversion will start complaining. I have no idea what revision control system is used by the SugarCRM developers—if they use any at all—but I would love to know how they handle these kinds of issues. Unfortunately the development process of SugarCRM is just as closed as that of Microsoft or Apple (something I commented on earlier) so we may never know.