In “Keeping SugarCRM under Subversion control” I showed you how I maintain the base SugarCRM in the face of changes to the core code. The setup described in that article allows me to make changes to the core code without running into (too much) trouble when SugarCRM ships a new version. It does have one downside: You cannot use Studio to make any changes. Instead, I keep all my customization work in a separate installable package, together with any custom modules I develop.
In this article I will show you how I develop my custom modules, how I keep them in Subversion and how they work together with the base SugarCRM from the previous article.
0: Table of contents
- 1: Big design up front
- 2: Importing the package in Subversion
- 3: Building installation packages
- 4: Workflow
- 5: Conclusions
1: Big design up front
In an earlier article of mine named “Fixing one-to-many relationships in SugarCRM 5.1” I commented on the quality of the Module Builder. It works okay-ish but there are a lot of bugs in it. Many of these can be fixed quite easily with a dash of PHP, but after that you cannot use the Module Builder to work on your module anymore. This means that I need to design my custom modules up front. I figure out all the modules that I need, all the relationships between them, all the fields that need to be added, etcetera. After that I create as much as possible in the Module Builder. I do this in a checked out branch of SugarCRM.
- svn copy http://svn.example.org/sugar/trunk \
- http://svn.example.org/sugar/branches/design \
- -m "Creating a branch to design the custom modules"
- cd /var/checkouts
- svn checkout http://svn.example.org/sugar/branches/design
When I am done building in the Module Builder I use “Publish” to create an installable module zip for me. Then I add the custom/modulebuilder directory to my Subversion branch and commit it. This way I can always get back to the original module prior to my customizations.
- svn add custom/modulebuilder
- svn commit -m "Finished designing my custom modules"
2: Importing the package in Subversion
I am assuming that you have a new, empty repository for the package available at https://svn.example.org/sugar-package. This is where I will import the unzipped version of the installable package that I generated in the last section. Lets create the bare repository layout first.
- cd ~
- mkdir -p temp/branches import/tags import/trunk
- svn import temp http://svn.example.org/sugar-package
Next, make a checkout of the trunk, add a build and src directory and put the contents of the package zip file in the src directory.
- svn checkout http://svn.example.org/sugar-package/trunk
- cd trunk
- svn mkdir build
- svn mkdir src
- cd src
- unzip ~/MyPackage2008_12_01_143123.zip
- svn add *
- svn commit -m "Initial import of MyPackage"
3: Building installation packages
From here on you could create a fresh install package by simply zipping the contents of the src directory, but I recommend against that. The zip will now also contain the hidden .svn directories that you really want to strip out. I use (g)vim for all my editing which creates .swp files when a file has been opened for editing. These need to be stripped out as well. Plus there is the versioning to consider. SugarCRM will attempt not to install a package when it sees that this version has already been installed, so I want to manually increase the version number as well.
Sounds like a job for a good bash script or even a Makefile. I opted for creating a bash script. It does a couple of things. First it checks the local revision number against the latest revision on the Subversion server. If your working copy is out of date it will ask you if it should run “svn update” for you. It also warns you if you have uncommitted changes in your working copy. Then it creates the installable zip file while stripping out all .svn directories and .swp files. It also sets the correct version number and date in the manifest.php file, based off the current Subversion revision number.
In the manifest.php file I marked the place where the version number and date should be put as follows:
- <?php
- $manifest = array (
- ...
- 'published_date' => '@DATE@',
- 'version' => '@VERSION@',
- );
- ...
- ?>
And here is the buildpackage.sh script. I put it in the trunk directory and added it to the repository. You can download buildpackage.sh as well.
- #!/bin/bash
- ##############################################################################
- #
- # buildpackage.sh
- #
- # Build an installable Sugar package $PACKAGE from the $SRCDIR directory.
- # Written by Sander Marechal <s.marechal@jejik.com>
- # Released into the Public Domain
- #
- ##############################################################################
- PACKAGE="TIM"
- SRCDIR="src"
- BUILDDIR="build"
- STAMP=`date '+%Y%m%d%H%M%S'`
- DATE=`date --rfc-3339 seconds`
- # find the local and remote revision numbers
- LOCALREV=`svn info -R $SRCDIR | sed -n 's/Revision: \([0-9]\+\)/\1/p' | sort -ur | head -n 1`
- REMOTEURL=`svn info $SRCDIR | sed -n 's/URL: \(.*\)/\1/p'`
- REMOTEREV=`svn info $REMOTEURL -R | sed -n 's/Revision: \([0-9]\+\)/\1/p' | sort -ur | head -n 1`
- PACKAGEDIR=$BUILDDIR/$PACKAGE-$STAMP
- VERSION=r$LOCALREV
- ZIPFILE=$PACKAGE-$VERSION.zip
- svn_update() {
- read UPDATE;
- case "$UPDATE" in
- [yY]*|"")
- svn update;
- LOCALREV=`svn info $SRCDIR | sed -n 's/Revision: \([0-9]\+\)/\1/p'`
- ZIPFILE=$PACKAGE-r$LOCALREV.zip
- ;;
- [nN]*)
- ;;
- [aAqQ]*)
- exit 0;
- ;;
- *)
- echo -n "Please enter [Y]es, [n]o or [a]bort: ";
- svn_update
- ;;
- esac
- }
- keep_local_changes() {
- read KEEPCHANGES
- case "$KEEPCHANGES" in
- [yY]*|"")
- VERSION=$VERSION+$STAMP
- ZIPFILE=$PACKAGE-$VERSION.zip
- ;;
- [nNaAqQ]*)
- echo "Please commit your changes first";
- exit 0;
- ;;
- *)
- echo -n "Please enter [Y]es or [n]o: ";
- keep_local_changes
- ;;
- esac
- }
- if [ "$LOCALREV" -lt "$REMOTEREV" ]; then
- echo "Local working copy seems out of date. Working copy is at r$LOCALREV but HEAD is at r$REMOTEREV";
- echo -n "Do you want to run 'svn update' [Y/n/a]? ";
- svn_update
- fi
- # Check for local changes
- CHANGES=`svn status $PACKAGE | grep -v "^\?" | wc -l`
- if [ "$CHANGES" -gt 0 ]; then
- echo -n "Local working copy has uncommitted changes. Continue [Y/n]? ";
- keep_local_changes
- fi
- # Create the build directory
- if [ ! -d "$BUILDDIR" ]; then
- mkdir $BUILDDIR;
- fi
- # Copy package to the build directory
- mkdir $PACKAGEDIR;
- cp -r $SRCDIR/* $PACKAGEDIR/;
- # Remove all the .svn dirs
- SVNDIRS=`find $PACKAGEDIR -name ".svn"`
- for SVNDIR in "$SVNDIRS"; do
- rm -rf $SVNDIR;
- done
- # Remove all the .swp files from Vim
- SWPFILES=`find $PACKAGEDIR -name "*.swp"`
- for SWPFILE in "$SWPFILES"; do
- rm -f $SWPFILE;
- done
- # Replace @VERSION@ and @DATE@ in the manifest
- sed -e "s/@VERSION@/$VERSION/g" -e "s/@DATE@/$DATE/g" $PACKAGEDIR/manifest.php > $PACKAGEDIR/manifest2.php
- rm -f $PACKAGEDIR/manifest.php
- mv $PACKAGEDIR/manifest2.php $PACKAGEDIR/manifest.php
- # Create the zip file
- if [ -f $ZIPFILE ]; then
- rm -f $ZIPFILE;
- fi
- cd $PACKAGEDIR
- zip -qr ../$ZIPFILE .;
- cd ../..;
- # Clean the build directory
- rm -rf $PACKAGEDIR;
- # All done
- echo "Succcesfully built package $BUILDDIR/$ZIPFILE";
- exit
4: Workflow
Using this package system, my workflow is pretty easy. I keep two checkouts of the base SugarCRM code. One is in /var/checkouts/trunk and one in /var/checkouts/work. I keep the trunk clean and use the work version to mess around with. When I code, I make changes to the package files, use the buildpackage.sh script to generate an installable zip and then I install that on the work checkout. You can simply install it over the older package because the build script has increased the version number. Occasionally I also need to run “Quick rebuild and repair” after that, but only if I changed the vardefs to create new fields that need to be stored in the database. When I need to make changes to the core files I do that to the trunk, commit it and then I run “svn update” on the work version.
All my customizations go in the installable package. I don't add anything to the base SugarCRM repository, not even upgrade-safe changes. When I need to upgrade a deployed version, I first upgrade the database and base SugarCRM code and then I install the updated package on top of it.
5: Conclusions
This workflow has not caused me any problems yet. I get to make any changes I want and keep everything under version control. The only downside, as explained in my previous article, is that I cannot use Studio anymore to add custom fields and make layout changes. Luckily, after a few tries it becomes really easy to make these changes directly in the viewdefs and vardefs source code.
I hope that these two articles will help you get your Sugar customizations under control. Happy hacking!
Comments
#1 Sam
Thank you for sharing your experience with us.
Do you know if this method is working with CVS instead of SVN ?
I don't know if SVN create some directories but CVS does (CVS folders in each folder). And then some functionalities doesn't work anymore (like Studio for example).
I think that every module that uses directories "introspection" is not working with a CVS checkout.
Any idea ?
thanx
Sam
#2 Sander Marechal (http://www.jejik.com)
But you can work around it if you really want to. Simply make sure that your webserver cannot read the subversion or CVS directories.
#3 michel.d
First, thanks for these advices.
Maybe I didn't get the entire process, but we still encounter one problem if we follow what you suggest, as the manifest.php generated when exporting our module will ignore some files contained in the following folders (although these are included in the zip archive):
- tpls
- views
- Ext
- ...
Moreover, we added new folders in the custom directory, according to the developer's guide (Charts, Ext), and these are not included in the zip file when exporting a custom package (this is corrected when using a script like your does).
We found out that the manifest.php file only takes language and metadata folders, so if we want to keep on using the import function in the interface, we'll have to modify the generation of manifest.php file, don't we?
If you any other solution, i would be really interested in.
Thanks by advance.
#4 Sander Marechal (http://www.jejik.com)
Note that this will give you just the custom module that you designed. This does not export all customisations that you made in the custom/ directory. If you want you can add such changes to the package manually as well. Instead of creating and modifying files in custom/, add those files to the package and use the "copy" directives in the manifest to copy them to the correct location when you install it.
The goal is to put all your customisations in the package and never make any customisations to SugarCRM directly if you can avoid it. It should be possible to setup a fresh, new installation of SugarCRM, install your package and have all your customisations ready.
#5 michel.d
We have to customize existing modules and create new modules.
Our goal is to be able to deliver customizations in an installable package in a new SugarCRM, but also on our production application, which has has already been plugged with earlier versions of our customizations.
We can't just use the new package and have to get nearly the whole custom/ directory in our installable package. So we'll have to create our own manifest file to include all our customizations (as they say we have to do it for new Dashlets or Themes in the Developper guide)
Thx!
#6 Sander Marechal (http://www.jejik.com)
#7 dasho
Is it possible to publish a package from the command-line, without using the button "Publish" of the ModuleBuilder?
These could be useful in order to apply quickly the latest modifications.
#8 Sander Marechal (http://www.jejik.com)
As far as I know there is no way to "publish" something that you create in the module builder from the commandline. Then again, why would you want to do that? If you're building in the module builder GUI already then you can publish from there as well. If you build purely from scratch then you don't need the module builder and you don't need to publish at all.
#9 dasho
Time after time I would like to check what I have built so far, how it looks like, what needs to be fixed, etc. In order to do this, I have to "publish" the package, send the zip file to the web server, merge the changes with the src of the package in the svn, run the script buildpackage.sh, get the zipped package from the web server, install it from the GUI to the testing copy of the application. Finally, check how the latest version looks like.
This is tedious and makes it very difficult to check small changes quickly. I would prefer something like this: run a script on the web server and get the published version from the version of the application that is is used for building, run another script to merge it with the src of the package in subversion, run buildpackage.sh, run another script to install the resulting zip to the testing version. Then, probably I could combine all the scripts in a single one: apply.sh . So, I could do some building, run apply.sh and check how the new changes work.
But maybe this is too idealistic.
#10 Sander Marechal (http://www.jejik.com)
My recommendation is to simply map out your package and draw up a simple ERD that contains all the beans (modules), their fields and the relationships between them. Don't worry if you don't get it 100% correct. Build everything in the Module Builder, Publish, unzip, put the source under version control and never, ever go back to the module builder again.
The time you need to manually add/edit fields and relationships directly in the source that you forgot when you were using the module builder is far less than the time you need trying to merge the module builder changes back into your existing source code.
#11 dasho
However, having to implement a long list of new modules, and having some time constraints, I decided to break the "Big design up front" into several chunks. So, I divided the list of new modules into several groups, where the modules of each group are somehow related to each-other. Then I planned several milestones for the implementation of the project: milestone_1 will implement the first group of modules, milestone_2 the second group, and so on.
For the implementation, I made a "Big design for the milestone_1" in Module Builder, creating the new modules, relationships, fields, etc.; in short as much as possible. Then I published the package and continued with the steps that you describe to import it into subversion. Then I ran the script buildpackage.sh and installed the new package into the working copy of SugarCRM.
I continued testing and refining these packages until the packages were fully functional, until everything worked fine and until milestone_1 could be called finished.
By the way, it was very easy to customize the package manually and to check immediately how it looks like. Why? Because I applied the modification first on the working copy of the application, and after I checked that it worked well, I applied it on the package as well. Seems like double work, however for small modifications it is OK. It avoids having to rebuild and reinstall the package just to check a small modification, and the development process becomes a bit more "incremental".
After I am done with milestone_1, I go back to the Module Builder and start building the modules of the milestone_2: creating them, creating the relations between them, adding new fields, and trying to build as much as possible. Then I publish the package again, and (here comes the difficult part), I unzip it and try to merge this new package with the package that I have already imported into the subversion. In order to facilitate the merge, I try to be careful in the Module Builder so that I don't touch at all the modules that were built during the milestone_1, or at least to modify them as little as possible (just any relationship, if necessary, and nothing else). After that, I don't go back to the Module Builder again, until the milestone_2 is finished and working correctly and the time comes to start with milestone_3.
So, in general, there is SugarCRM, with new releases coming from vendor_1 time after time, which I have to patch and fix. But there is also a sugarcrm_package, built with Module Builder, with new releases coming from "vendor_2" time after time, which also needs to be fixed and patched. All the customizations go to the sugarcrm_package. The application is built by installing the patched SugarCRM, and then installing on it the patched sugarcrm_package.
The truth is that right now I have just finished the milestone_1, and I have not started yet with the milestone_2, so I am not sure how well it is going to work. However I think that it should work. Dividing the big design into milestones helps to make the development a bit more iterative. In my opinion, "iterative and incremental" is the best approach for building an application, whenever it is possible.
#12 Sander Marechal (http://www.jejik.com)
#13 dasho
Anyway, I haven't tried yet one approach or the other.
Meanwhile, I have started to document the things that I am doing for developing my application. It is not finished yet, but still you can have a look and give me any idea:
http://sugarcrm.iskey.info
#14 dasho
I have discovered a discussion that shows how to deploy the modified code of the package from the command line, without having to build the zip file and to install it from the Module Loader:
http://www.sugarcrm.com/forums/showthread.php?p=196944
It can be useful for testing the package modifications quickly. I have modified it a bit to adopt it for version 5.5.0:
#!/usr/bin/php
<?php
/**
This utility uploads a module located outside the sugarCRM tree
you are currently developing. This is bypassing the zipfile and
the module interface and makes easy the development of a module.
It outputs the error created by php to stderr, so you can immediately
see what's wrong.
*/
//set this to your MODULE DEVELOPMENT directory
//it is the directory that contains the file 'manifest.php'
$package_dir = "/var/www/sugar_packages/pkg_name/";
//set this to the SugarCE installation where the package will be deployed
//it is the directory that contains 'index.php'
$sugar_dir = "/var/www/p550t/";
echo "\nPackage dir is set to: $package_dir";
echo "\nSugar dir is set to : $sugar_dir\n";
//go to the top of the SugarCE directory
chdir($sugar_dir);
//make sure we dump the stuff to stderr
ini_set("error_log","php://stderr");
//initialize
if(!defined('sugarEntry')) define('sugarEntry', true);
require_once('include/entryPoint.php');
require_once('ModuleInstall/ModuleInstaller.php');
$current_user = new User();
$current_user->is_admin = '1';
//initialize the module installer
$modInstaller = new ModuleInstaller();
$modInstaller->silent = true; //shuts up the javscript progress bar
//start installation
echo "\nStarting...\n";
$modInstaller->install($package_dir);
echo "\n\nDone.\n";
?>
However, when the package is ready, I still have to build the zip file and install it through the web interface of the Module Loader.
I have also tried this script to install the zip file from the command line:
#!/usr/bin/php
<?php
/**
This utility installs a package zip file from the command line,
without using the web interface.
*/
//set this to the SugarCE installation where the package will be deployed
//it is the directory that contains 'index.php'
$sugar_dir = "/var/www/p550t/";
//check the arguments
if($argc != 2)
{
echo "\n\nUSAGE: ./package_install.php <package_file.zip>\n";
exit;
}
//get the full path of the package file
$package_file = dirname(__FILE__).$argv[1]
echo "\nSugarCE dir is set to: $sugar_dir";
echo "\nPackage dir is set to: $package_file\n";
//go to the top of the SugarCE directory
chdir($sugar_dir);
//make sure we dump the stuff to stderr
ini_set("error_log","php://stderr");
//initialize
if(!defined('sugarEntry')) define('sugarEntry', true);
require_once('include/entryPoint.php');
require_once('ModuleInstall/PackageManager/PackageManager.php');
$current_user = new User();
$current_user->is_admin = '1';
//initialize the module installer
$pkgManager = new PackageManager();
//start installation
echo "\nStarting...\n";
$pkgManager->performSetup($package_file, 'module', false);
$uploaded_file = $sugar_config['upload_dir']
. "/upgrades/module/" . basename($package_file);
$pkgManager->performInstall($uploaded_file);
echo "\n\nDone.\n";
?>
However, somehow, it installs the new version of the package over the old one, without replacing it.
#15 treats
I took over the Sugar development at my company and we had already used Studio to make modifications. We planned a major release with a handful of new modules and relationships between these custom modules and the core modules. I did build all of these as packages in the Module Builder -- this way I would be able to easily migrate this to a clone of production to test my work. Good so far, less two issues.
1) One-to-many relationships from stock modules to custom modules. I couldn't find a way to do this within module builder so I ended up doing this customization within Studio after the fact.
2) When issues started getting logged against this release I was fixing them with Studio -- bad.
My questions are these:
On a system that has been historically modified using Studio, is there a way to adapt this for version control?
How do you create one-to-many relationships from stock modules to custom with Module Builder (I did read your article about fixing one-to-many relationships but it appeared to only address the case where both modules are custom)?
Closing thoughts: I wish I could go back and not use Studio... Nay, I wish Studio would dictate code not the DB!
#16 Sander Marechal (http://www.jejik.com)
As for many-to-one relations to standard modules, see this article.
#17 Jeremy Ehrenthal (http://echealthinsurance.com/)
#18 kamlesh Dhayal
I spend 6-8 hr on your site to learn many-to-one relationship in SugarCRM. i got you point but i have some question in my mind.
I am using SugarCRM on Window SP2 and i don't know about SVN.
How to run your buildpackage.sh on windows ?
Can i need to install SVN Enviroment on my machine ?
if yes then tell me some step to install and to run your file on window.
Any idea ?
Thanx
Kamlesh D
#19 Sander Marechal (http://www.jejik.com)
The buildpackage.sh script is designed for Linux. It does not work on Windows. You could try to run it using Cygwin though, but I am not sure that will work. Cygwin gives you a unix/linux-like shell on Windows.
I hope this helps!
#20 grant.k
How would you migrate the data from the Studio days into the new SVN world? Would this require custom migration scripts or can it be done via Sugar export/import?
#21 Sander Marechal (http://www.jejik.com)
A custom migration script is probably the fastest way to do this. It can be as simple as a couple of "INSERT INTO ... SELECT ..." statements.
#22 grant.k
#23 SugarCRM Integration (http://phelpsaron.hpage.co.in/home_80726412.html)
Comments have been retired for this article.