Working with live and development sites

RedFox · 2011-09-20 11:11:52

Sometimes I make a dev section with page and template structure (live site) so I can experiment without to much risk with HTML/Javascript/CSS/articles/etc. This section is a standalone part of the site … with it’s own url so only invited people are welcome. You can even lock the landing page if you wish. And … don’t forget to block Google … ;-)

[edit]
Oops … just saw Bloke’s post and others … :|

Last edited by RedFox (2011-09-20 11:22:18)

hakjoon · 2011-09-20 14:57:54

I’ve been toying with the idea of creating an XML-RPC client plugin which would allow moving pieces between different TXP installations. The problem is that the use of auto-increment IDs everywhere makes that tricky because the IDs might not match across the environments.

I’ve thought of maybe changing the tables on the higher level environment sites to not auto-increment, but that would force a workflow of no work on live (which could be a good thing). Haven’t looked into other ways of dealing with this in MySQL otherwise.

Wasn’t sure if there was any interest in this so I haven’t been super pursuing it, but I started thinking about it as a way to have clients stage content updates to existing copy.

caruso_g · 2011-09-25 09:33:43

Hello Txp People(s)

We currently have a production setup as follows:

A production instance
A test instance

We do development on the test instance (yes: perhaps we should rename that. Hmm). We’ve run into a problem whereby our existing migration process, copying the DB + files, would end up overwriting data on the production instance (e.g: it’d delete it because it is never copied back to test).

An example would be: We post up new blog entries on production, but these don’t come across to test. We don’t want to use a “copy” process to update test from production, as it’ll overwrite existing new/modified development on the test instance (e.g: forms held inside the DB).

In our case it is only blog articles that are effected. We’ve been considering just putting a trigger onto the DB to inserts rows into the test instance textpattern table as they are inserted into the production instance (the DB schemas are on the same server).

Is this stupendously stupid? In your experience, have you come across this and found a better way? As long as MySQL auto-inc columns play ball with the trigger (I’ve not tried it yet) I’m presuming that we’ll be able to then simple overwrite the production instance with the test instance (along with any changed forms, css or js files, images etc).

I figured it’d be smart to first ask if anyone has tried this kind of thing before in a txp context and if it really just asking for trouble.

Thanks in advance,
Giuseppe

(I merged the new topic started by caruso_g with this existing topic, trying to keep all info on the subject together. Hope it doesn’t get too confusing. -Els)

Last edited by els (2011-09-25 23:14:24)

Algaris · 2011-09-25 12:15:12

That’s an interesting idea hakjoon. If the mechanism for moving content between the installations was fairly easy/seamless it could be what I’m looking for, till something is baked into TXP5 ;-)

The only drawback I can think of is it would make updating the site I work on impossible when out of the office.

els · 2011-09-25 19:34:54

~~Hi Giuseppe, have you seen this recent thread? Maybe it helps, it’s more or less about the same issues.~~

Edit: topics merged.

Last edited by els (2011-09-25 23:15:27)

caruso_g · 2011-09-25 22:04:59

Els wrote:

Hi Giuseppe, have you seen this recent thread? Maybe it helps, it’s more or less about the same issues.

Hi Els, thanks so much for the link, I didn’t see it.

More or less, it is the same issue. We are really trying to find a solution since working on live sites it’s not a viable way on more complex site in any case.

Currently, we are testing a possible solution considering that the only content created in Production are blog articles.
We created a MySql trigger function to mirror articles created in Production to Development. Then we created a script to migrate Development to Production when ready.
We are also considering to use cnk_versioning to manage structure so that we’ll just move its folder structure from Development to Production after we tested edits to migrate.

What do you think about it?
Do you think cnk_versioning could help in keeping structure more separated from content¹?
Which TXP content/folders can be centralized to ease migrations? Images folder? Css folder?
Could a multi-site installation help in any way?

Thank a lot as always for your help, it is much appreciated. :)

¹ btw, keeping all in texpattern table doesn’t help…

Edit:
Added MySql trigger functions reference link

Last edited by caruso_g (2011-09-25 22:11:17)

caruso_g · 2011-09-25 22:10:03

Hi, I am sorry, I didn’t see this post and I opened a new thread in “How do I…?” section.
Please, take a look at it , I hope that can help.

(Topics were merged. -Els)

Last edited by els (2011-09-25 23:17:20)

hakjoon · 2011-09-26 01:49:07

Algaris wrote:
The only drawback I can think of is it would make updating the site I work on impossible when out of the office.

Yeah the ideal situation would be to be able to move things between any installation that way you could work on live if needed and migrate the stuff back, or work on test or dev. It only really gets tricky when using ids to reference content like images, files or specifying articles by id. It would be nice to move to guids or something to alleviate that but that brings it’s own issues.

Still trying to think through it to see what can be done. I’ve worked with stuff like this before but we always targeted multiple environments from the get go so the DB was setup to handle that.

scornflake · 2011-09-26 19:33:30

hakjoon – I’m curious. What issues do you foresee moving to a guid (or simply synthetic ID) based approach?

.

Algaris · 2011-10-12 08:24:21

In order to try and simplify my updating process until Sam’s Escher’s branch features (hopefully) become part of Textpattern 5 I’ve been looking at different options.

Currently when working with my local and remote installations of TXP I try to keep all articles, images, files and links in sync with each other so as to avoid any possible confusion with ID numbers. Typically I’ll use the local version as my master. When ready to push major updates out I’ll make back ups of both the local and remote database and FTP my Images and Files directories to the remote site. I then drop the remote database, import the local database and change the file locations in Textpattern’s preferences. Normally this process works fine and only takes a minute or two but it’s not a lot of fun and can be pretty hairy (I really don’t like messing around with the databases). The fact that the whole website is offline or appears broken to visitors during this time is another reason I’m not happy with the process.

Recently I came across this software http://www.zennaware.com/cornerstone/index.php which uses Subversion. Is this something that I could use to push updates to the remote server? Given that TXP stores its Pages, Forms and Articles, etc. in a mySQL database I’m not sure how easily it could be synced and I would still have to manually update my TXP preferences for the file locations.

jakob · 2011-10-12 09:40:42

While using svn for versioning files is fine, I don’t think you can use svn to sync MySQL databases (except for db-dumps), but I could be mistaken.

You can slightly simplify your db transfer by dropping and replacing only individual tables of the databases so that you don’t have to reset your prefs. Alternatively use adi_prefs to store local and remote prefs sets and switch between them or an sql dump of just your prefs which you can use to reinstate them after a db-update.

From googling I found two projects that attempt to address the problem of migrating mysql databases:

Maybe others can provide a more qualified reply or these links are useful pointers for further investigations.

Algaris · 2011-10-12 10:27:05

Thank you for your reply. Well I guess SVN might be useful to keep files, images and my JS folder in sync between the local and remote servers. It’s a pity (but not unexpected) regarding mySQL syncing. I wonder if I could set something up where I export my local database (using adi_prefs for the remote preferences), upload the db-dump with the rest of my site via SNV and then import the db-dump using rss_admin_db_manager to avoid dropping tables and any downtime.

Ideally what I’d love would be a one click solution where my database is pushed to the remote site with all the correct preferences, no messing around with import/export, dropping tables and most importantly no downtime.

Algaris · 2011-11-23 10:07:14

After some major headaches trying to keep our Development and Live sites in sync I finally caved in and set up new Development and Staging copies of the website.

I updated the website last night using this method and it almost had me longing for our old website which was updated using Dreamweaver. If I wanted to make a structural change I downloaded the file to a localhost copy of the site, edited it and then pushed it back to the live site all via Dreamweaver’s file management window. Two clicks (one to download, one to upload) and that was it the changes were live.

With Textpattern if I want to work locally I have to:

1. Make the changes to the development copy of the website.
2. Copy the Live site to the Staging site.
3. Export the CSS, Forms and Pages tables from the Development site.
4. Drop the CSS, Forms and Pages tables from the Staging site.
5. Import the Development CSS, Forms and Pages tables into the Staging site.
6. Test the Staging site.
7. Fix any bugs on the Development site.
8. Repeat steps 3-5.
9. Export the CSS, Files, Pages tables from the Staging site.
10. Enable rvm_maintenance on the Live site.
11. Drop the CSS, Files, and Pages tables from the Live site.
12. Import the Staging CSS, Files, and Pages tables into the Live site.
13. Test the updated live site (and hope that nothing went wrong)
14. Disable rvm_maintenance.

There has to be an easier way to work than this. I’m dreading the day I have to rush out an emergency fix. I guess I’ll have to bite the bullet and work straight on the live site for this, which I dislike doing for numerous reasons.

After going through this process I really hope that Sam’s Branch feature from Escher is made part of the Textpattern 5 core. It’s number one on my Textpattern wish list right about now.

jakob · 2011-11-23 11:53:27

Maniqui has come up with a somewhat elaborate but rather clever system for a few of us on the forum who have been working together. The basic principle is as follows but maniqui can elaborate in detail.

We’re using cnk_versioning throughout so that the forms, pages and css can be stored in a file versioning system. In this particular case it’s git-based with a central repository on beanstalk but the same principle should work with other cvs systems.
The main online repository is essentially the staging / develop site and is hosted with beanstalk.
The live site is a remote copy of the staging site that is updated manually by pulling across the changes from the staging site.
Each person working on the site has a local installation in a virtualhost on their computer linked to their git-monitored local working copy of the repository.
Certain config files specific to the local / staging / live sites are excluded from the versioning process.

It takes a bit of setting up, and maniqui has it set up even more cleverly than I can explain, but once it is in place, each person involved on working on a site has the same set of files on the computer and staging site. You can develop and test locally, then commit the changes you make to the online repo (the staging site). Here you can coordinate with others and when everything’s been tested to completion, the changes are then pulled across to the live site.
You could also use the branching possibilities offered by git and co to keep multiple branches of the same repo on the go (e.g. develop, live, new-feature-x, new-feature-y, bugfix) and then merge these together when the new feature has been completed or the urgent bug is fixed. Most of this was born out of the need for several people to work on a single site, so it can probably be simplified somewhat for a one-person operation.

What isn’t versioned at present is the DB, though. adi_prefs or a small custom block of importable sql makes it possible to export/import the db and then overwrite specific tables with local setup config information (alternatively don’t import/export all the tables).

———

A poor man’s alternative: for my own stuff I don’t generally need an interim staging site, but do work from different computers (at work / at home). I use Espresso as my preferred code editor (IIRC you do too) and stick my template files and some other key files such as css/js etc. into a folder that is synced via DropBox between my machines. That folder can be the basis for my local virtualhost container, or you can “plug” these directories via a symlink to your own virtualhost container elsewhere in your file system.

Espresso allows you to attach several online ftp servers to a single site and control when saving a file syncs to the online repo or not. You can develop locally just by saving the files. When you’re ready to upload the changes, switch on sync with the online server and either re-save the files or use Espresso’s sync function (depending on how much you changed). As you can attach more than one server, you could theoretically add a staging ftp and a live ftp to your local Espresso project. You do need to leave your online site on “testing” production status, though.

Algaris · 2011-11-23 12:15:44

Wow thank you for this post jakob. It’ll take me some time to ruminate over what you’ve said. I definitely would be interested in reading more from maniqui regarding this setup.

Currently I find designing a website in Textpattern as well as updating it with new content to be a pure joy. And I love the community so much. Maintaining a development copy however is a pain in the ass.

You remember correctly I do use Espresso. Currently my setup is:

cnk_versioning
Espresso
LESS.app (http://incident57.com/less/) I will however be upgrading to Code Kit at some point (http://incident57.com/codekit/)
Transmit for pulling content from the Live site to the Staging site (I love being able to mount an FTP server as a disk)

All of my development work is done on an internal server where I work which unfortunately isn’t accessible to the outside world.

I have been looking at versioning. I started with a trial of Cornerstone a little bit back but recently stumbled a cross http://www.git-tower.com I have to admit I’d rather use a client on my Mac for this kind of thing. I still have a way to go as I don’t fully understand the all concepts regarding versioning yet.

Textpattern CMS

Textpattern CMS support forum

#16 2011-09-20 11:11:52

Re: Working with live and development sites

#17 2011-09-20 14:57:54

Re: Working with live and development sites

#18 2011-09-25 09:33:43

Re: Working with live and development sites

#19 2011-09-25 12:15:12

Re: Working with live and development sites

#20 2011-09-25 19:34:54

Re: Working with live and development sites

#21 2011-09-25 22:04:59

Re: Working with live and development sites

#22 2011-09-25 22:10:03

Re: Working with live and development sites

#23 2011-09-26 01:49:07

Re: Working with live and development sites

#24 2011-09-26 19:33:30

Re: Working with live and development sites

#25 2011-10-12 08:24:21

Re: Working with live and development sites

#26 2011-10-12 09:40:42

Re: Working with live and development sites

#27 2011-10-12 10:27:05

Re: Working with live and development sites

#28 2011-11-23 10:07:14

Re: Working with live and development sites

#29 2011-11-23 11:53:27

Re: Working with live and development sites

#30 2011-11-23 12:15:44

Re: Working with live and development sites

Board footer