Working with live and development sites

Algaris · 2011-10-12 08:24:21

In order to try and simplify my updating process until Sam’s Escher’s branch features (hopefully) become part of Textpattern 5 I’ve been looking at different options.

Currently when working with my local and remote installations of TXP I try to keep all articles, images, files and links in sync with each other so as to avoid any possible confusion with ID numbers. Typically I’ll use the local version as my master. When ready to push major updates out I’ll make back ups of both the local and remote database and FTP my Images and Files directories to the remote site. I then drop the remote database, import the local database and change the file locations in Textpattern’s preferences. Normally this process works fine and only takes a minute or two but it’s not a lot of fun and can be pretty hairy (I really don’t like messing around with the databases). The fact that the whole website is offline or appears broken to visitors during this time is another reason I’m not happy with the process.

Recently I came across this software http://www.zennaware.com/cornerstone/index.php which uses Subversion. Is this something that I could use to push updates to the remote server? Given that TXP stores its Pages, Forms and Articles, etc. in a mySQL database I’m not sure how easily it could be synced and I would still have to manually update my TXP preferences for the file locations.

jakob · 2011-10-12 09:40:42

While using svn for versioning files is fine, I don’t think you can use svn to sync MySQL databases (except for db-dumps), but I could be mistaken.

You can slightly simplify your db transfer by dropping and replacing only individual tables of the databases so that you don’t have to reset your prefs. Alternatively use adi_prefs to store local and remote prefs sets and switch between them or an sql dump of just your prefs which you can use to reinstate them after a db-update.

From googling I found two projects that attempt to address the problem of migrating mysql databases:

Maybe others can provide a more qualified reply or these links are useful pointers for further investigations.

Algaris · 2011-10-12 10:27:05

Thank you for your reply. Well I guess SVN might be useful to keep files, images and my JS folder in sync between the local and remote servers. It’s a pity (but not unexpected) regarding mySQL syncing. I wonder if I could set something up where I export my local database (using adi_prefs for the remote preferences), upload the db-dump with the rest of my site via SNV and then import the db-dump using rss_admin_db_manager to avoid dropping tables and any downtime.

Ideally what I’d love would be a one click solution where my database is pushed to the remote site with all the correct preferences, no messing around with import/export, dropping tables and most importantly no downtime.

Algaris · 2011-11-23 10:07:14

After some major headaches trying to keep our Development and Live sites in sync I finally caved in and set up new Development and Staging copies of the website.

I updated the website last night using this method and it almost had me longing for our old website which was updated using Dreamweaver. If I wanted to make a structural change I downloaded the file to a localhost copy of the site, edited it and then pushed it back to the live site all via Dreamweaver’s file management window. Two clicks (one to download, one to upload) and that was it the changes were live.

With Textpattern if I want to work locally I have to:

1. Make the changes to the development copy of the website.
2. Copy the Live site to the Staging site.
3. Export the CSS, Forms and Pages tables from the Development site.
4. Drop the CSS, Forms and Pages tables from the Staging site.
5. Import the Development CSS, Forms and Pages tables into the Staging site.
6. Test the Staging site.
7. Fix any bugs on the Development site.
8. Repeat steps 3-5.
9. Export the CSS, Files, Pages tables from the Staging site.
10. Enable rvm_maintenance on the Live site.
11. Drop the CSS, Files, and Pages tables from the Live site.
12. Import the Staging CSS, Files, and Pages tables into the Live site.
13. Test the updated live site (and hope that nothing went wrong)
14. Disable rvm_maintenance.

There has to be an easier way to work than this. I’m dreading the day I have to rush out an emergency fix. I guess I’ll have to bite the bullet and work straight on the live site for this, which I dislike doing for numerous reasons.

After going through this process I really hope that Sam’s Branch feature from Escher is made part of the Textpattern 5 core. It’s number one on my Textpattern wish list right about now.

jakob · 2011-11-23 11:53:27

Maniqui has come up with a somewhat elaborate but rather clever system for a few of us on the forum who have been working together. The basic principle is as follows but maniqui can elaborate in detail.

We’re using cnk_versioning throughout so that the forms, pages and css can be stored in a file versioning system. In this particular case it’s git-based with a central repository on beanstalk but the same principle should work with other cvs systems.
The main online repository is essentially the staging / develop site and is hosted with beanstalk.
The live site is a remote copy of the staging site that is updated manually by pulling across the changes from the staging site.
Each person working on the site has a local installation in a virtualhost on their computer linked to their git-monitored local working copy of the repository.
Certain config files specific to the local / staging / live sites are excluded from the versioning process.

It takes a bit of setting up, and maniqui has it set up even more cleverly than I can explain, but once it is in place, each person involved on working on a site has the same set of files on the computer and staging site. You can develop and test locally, then commit the changes you make to the online repo (the staging site). Here you can coordinate with others and when everything’s been tested to completion, the changes are then pulled across to the live site.
You could also use the branching possibilities offered by git and co to keep multiple branches of the same repo on the go (e.g. develop, live, new-feature-x, new-feature-y, bugfix) and then merge these together when the new feature has been completed or the urgent bug is fixed. Most of this was born out of the need for several people to work on a single site, so it can probably be simplified somewhat for a one-person operation.

What isn’t versioned at present is the DB, though. adi_prefs or a small custom block of importable sql makes it possible to export/import the db and then overwrite specific tables with local setup config information (alternatively don’t import/export all the tables).

———

A poor man’s alternative: for my own stuff I don’t generally need an interim staging site, but do work from different computers (at work / at home). I use Espresso as my preferred code editor (IIRC you do too) and stick my template files and some other key files such as css/js etc. into a folder that is synced via DropBox between my machines. That folder can be the basis for my local virtualhost container, or you can “plug” these directories via a symlink to your own virtualhost container elsewhere in your file system.

Espresso allows you to attach several online ftp servers to a single site and control when saving a file syncs to the online repo or not. You can develop locally just by saving the files. When you’re ready to upload the changes, switch on sync with the online server and either re-save the files or use Espresso’s sync function (depending on how much you changed). As you can attach more than one server, you could theoretically add a staging ftp and a live ftp to your local Espresso project. You do need to leave your online site on “testing” production status, though.

Algaris · 2011-11-23 12:15:44

Wow thank you for this post jakob. It’ll take me some time to ruminate over what you’ve said. I definitely would be interested in reading more from maniqui regarding this setup.

Currently I find designing a website in Textpattern as well as updating it with new content to be a pure joy. And I love the community so much. Maintaining a development copy however is a pain in the ass.

You remember correctly I do use Espresso. Currently my setup is:

cnk_versioning
Espresso
LESS.app (http://incident57.com/less/) I will however be upgrading to Code Kit at some point (http://incident57.com/codekit/)
Transmit for pulling content from the Live site to the Staging site (I love being able to mount an FTP server as a disk)

All of my development work is done on an internal server where I work which unfortunately isn’t accessible to the outside world.

I have been looking at versioning. I started with a trial of Cornerstone a little bit back but recently stumbled a cross http://www.git-tower.com I have to admit I’d rather use a client on my Mac for this kind of thing. I still have a way to go as I don’t fully understand the all concepts regarding versioning yet.

Algaris · 2011-11-23 12:20:07

jakob wrote:

Espresso allows you to attach several online ftp servers to a single site and control when saving a file syncs to the online repo or not. You can develop locally just by saving the files. When you’re ready to upload the changes, switch on sync with the online server and either re-save the files or use Espresso’s sync function (depending on how much you changed). As you can attach more than one server, you could theoretically add a staging ftp and a live ftp to your local Espresso project. You do need to leave your online site on “testing” production status, though.

This is interesting. Because of the way cnk_versioning works all I’d have to do would be upload the files to the Pages, Form and CSS folders, refresh the Live site in my browser window and it would be online. Is that correct?

Bloke · 2011-11-23 12:21:20

I concur maniqui’s system works well — at least it does for those people working on proper operating systems! With me being stuck on Windows at the moment and everyone else being Mac/Linux-based, some of the filenames of the project were deemed “invalid” because of certain characters in the filenames when I synced the repo. Skipping those and being able to push to git proved problematic, so most of my edits I did “blind” on my local copy and then either:

published the changes to the Basecamp project so someone else could sync them on my behalf
uploaded the files directly via SFTP to the staging server
logged into the staging Txp installation / cpanel and made changes there

With the little script in place to poke local Txp prefs into the database it really is a breeze to keep a local dev copy in sync with a staging or live version of the database: just pull the latest DB down from a central location then double click the script to overwrite the prefs so I can use it in my XAMPP environment. It only alters about 5 or 6 pref values like filesystem paths and site URL.

Syncing back to live though is quite manual because — as you say — articles are going in and out of the DB from clients. Thus we need to cherry pick which tables get overwritten or exported from a local dev branch. Normally that’s not much of an issue because you only tend to work on a little bit at a time and can first push the changes to staging and — once approved — repeat the push to live. But if you’re working on a grand redesign rather than incremental alterations then it might require some rvm_maintenance downtime (cnk_versioning helps here, as jakob says).

Our rule of thumb when messing with the database: export anything in the Presentation section and leave the Content tables well alone unless you really know what you’re doing!

jakob · 2011-11-23 13:07:05

This is interesting. Because of the way cnk_versioning works all I’d have to do would be upload the files to the Pages, Form and CSS folders, refresh the Live site in my browser window and it would be online. Is that correct?

Yes, this is basically what I do myself when a staging site would be overkill, or when the staging site becomes the live site after development. If it’s just a quick fix, I’ll experiment with my local installation until it’s working how I want it, then flick on the switch to upload. When you next save it auto-syncs to the ftp without you having to ftp in a second step.

In Espresso 2 the switch is now next to the file and there’s an on/off slider and a drop-down that lets you choose the currently “active” remote server to sync with.

To set up additional remote servers, if you are in an Espresso project, choose “Settings” from the toolbar and you can add your server(s)’s credentials. Via the switcher in the screenshot above you can switch between them, so you could theoretically work locally, trial on your staging site, then switch the upload server to the live site and save/sync again. If you’ve changed a lot of files and don’t remember which files it affects, you can use the bulk sync function (in the left-hand bar at the bottom under the heading “publish” you’ll see your online servers. Click on that, then on sync in the toolbar / the cloud in the main window. You’ll see a sync review before it actually syncs and you can choose to skip files if you want. My only worry there is that it might be too easy to accidentally load stuff to the live site because Espresso doesn’t ask you constantly if you’re sure, so you need keep your wits about you.

You might find that if you work a lot on develop and then only periodically update the live site, that you can basically work on a local<->develop setup, and pull across the changes to the live site manually.

jakob · 2011-11-23 13:23:05

I have been looking at versioning. I started with a trial of Cornerstone a little bit back but recently stumbled a cross http://www.git-tower.com I have to admit I’d rather use a client on my Mac for this kind of thing. I still have a way to go as I don’t fully understand the all concepts regarding versioning yet.

I too am no command-line purist and use Versions for SVN and more recently, for the above project, Tower for Git (unfortunately each is native to its own versioning system and don’t work for both git and svn). Personally I like to see at a glance which files are affected, then click on the file and see the relevant diffs immediately, rather than typing in commands back and forth…

I did find it takes some getting used to and reading up on to get to grips with the versioning systems. Maniqui gave us some tutorials and tips to start with (there is actually quite a lot available online) and he has the most in-depth knowledge of the team. I get the impression that the GUI programs can’t do everything but cover the most common subset of functionality and anyway I think I have only scratched the surface of the GUI tools as yet too so I still feel like I have “learner-plates” on! You can still use the command-line, of course, and any changes are managed by git so reflect in Tower too.

Algaris · 2011-11-24 10:47:29

Thank you everyone there’s so much to think about.

@Bloke you need to get a real computer. Might I suggest a Mac ;-)

Bloke wrote:

With the little script in place to poke local Txp prefs into the database it really is a breeze to keep a local dev copy in sync with a staging or live version of the database: just pull the latest DB down from a central location then double click the script to overwrite the prefs so I can use it in my XAMPP environment. It only alters about 5 or 6 pref values like filesystem paths and site URL.

I’d be interested in knowing more about this script if possible.

jakob wrote:

Yes, this is basically what I do myself when a staging site would be overkill, or when the staging site becomes the live site after development. If it’s just a quick fix, I’ll experiment with my local installation until it’s working how I want it, then flick on the switch to upload. When you next save it auto-syncs to the ftp without you having to ftp in a second step.

I’m very tempted by this method. I’m all for simplicity and having things easy to manage.

However, I wonder whether going down a staging server and GIT route might be the smart option in the long run. For instance my boss asked me to do a major overhaul of several sections and navigation of the site at the request of the Senior Management. I did this modifying the development copy of the site. After completing the changes (and making some refinements to the site) I was told they had changed their mind and wanted to do something completely different. This left the development copy of the site unusable along with the other refinements that I’d made. Sure I could (and probably should) have done a mySQL dump and backed up all the files before starting but I can’t but help think that having a central repository that I can make branches from would be the smart move. That way I would be able to make minor incremental improvements and emergency patches while working on major restructuring.

Bloke · 2011-11-24 11:33:50

Algaris wrote:

you need to get a real computer. Might I suggest a Mac ;-)

Hehe, top of me list when I become rich :-D

I’d be interested in knowing more about this script if possible.

I can’t find my script right now, but here’s the generic approach. First, create a file called live_prefs.sql and inside it, put lines like this:

UPDATE txp_prefs SET val='/path/to/tmp' WHERE name='tempdir';
UPDATE txp_prefs SET val='/path/to/files' WHERE name='file_base_path';
UPDATE txp_prefs SET val='/path/to/site' WHERE name='path_to_site';
UPDATE txp_prefs SET val='site.com' WHERE name='siteurl';
...

and so on for each value that’s specific to the live installation. Other possible candidates are plugin_cache_dir, rss_dbbk_path (for rss_db_admin) and custom_scripts_path (for glz_cf). Just trawl through your prefs table looking for anything that might need altering.

Then create another file called dev_prefs.sql with the same SQL commands as above but with values for your dev environment.

Once you’ve done that you can then switch to dev by issuing this XAMPP command:

mysql -u username -ppassword database_name < /path/to/dev_prefs.sql

and of course back to live by piping in the live_prefs.sql file instead. You can put each command in a separate shell script, chuck ‘em on your desktop and just execute them with a double click, or depending on your scripting prowess you could get it to take a single parameter (live or dev) and use the same script to choose which file to use.

Once one of those is run your MySQL DB will be set to the relevant environment and you can either use it locally or upload it if it’s destined for the live server.

That’s the gist of it anyway. No rocket science involved! You can automate a lot of stuff this way to make life easier on yourself.

Textpattern CMS

Textpattern CMS support forum

#25 2011-10-12 08:24:21

Re: Working with live and development sites

#26 2011-10-12 09:40:42

Re: Working with live and development sites

#27 2011-10-12 10:27:05

Re: Working with live and development sites

#28 2011-11-23 10:07:14

Re: Working with live and development sites

#29 2011-11-23 11:53:27

Re: Working with live and development sites

#30 2011-11-23 12:15:44

Re: Working with live and development sites

#31 2011-11-23 12:20:07

Re: Working with live and development sites

#32 2011-11-23 12:21:20

Re: Working with live and development sites

#33 2011-11-23 13:07:05

Re: Working with live and development sites

#34 2011-11-23 13:23:05

Re: Working with live and development sites

#35 2011-11-24 10:47:29

Re: Working with live and development sites

#36 2011-11-24 11:33:50

Re: Working with live and development sites

Board footer