Working with live and development sites

maniqui · 2011-11-24 17:43:17

My nickname was mentioned more than 3 times, therefore, I have been summoned.

Yeah, I like toying with stuff, usually complicating things, and then, I just try to convince others to follow my methodologies :)

jakob and Bloke summed up pretty well some of the stuff I did for improving (or trying to) the development workflow with Textpattern.
These changes in workflow are still a WIP. The discussion we are having may help to improve things.

So, my motivations to begin tinkering with this stuff were:

to make developing websites with TXP more teamwork-friendly: putting stuff on filesystem (via cnk_versioning) really helps on this goal. Putting stuff on database doesn’t.
to make developing websites with TXP more (D)VCS-friendly (Git, SVN, etc): idem ^^.
to rely less:
- on DB for storing functionality & development-related stuff: TXP is, historically, “database-centric” (almost all stuff is stored in database). I used to love that. Now, not that much. I’d much prefer more stuff (particularly, plugin configs and other similar stuff) stored in files.
- on the web-based (TXP admin) interface for development tasks: because this usually means stuff stored in DB ^^.
to avoid doing (repeating) the same stuff on each new TXP project: for example, installing plugins on the DB is a no-no.

The changes I/we (jakob, Bloke, pieman) implemented in workflow helped to achieve & satisfy some of the motivations stated above.
I’d say there is still the “issue” of lot of development-related stuff (configurations, plugin settings, etc) being managed via TXP admin & stored in the DB (and I’m not talking about content) .

So, without going (yet) into details, these are some advices/tips for making it easier to work with TXP:

use cnk_versioning. Forget about using the Pages, Forms & Styles tabs (on Presentation tab).
version your code, even if you are doing work in solo-mode. May this be a great opportunity to learn some modern DVCS like Git (this website will really help you if you are a bit afraid: http://gitimmersion.com/).
run plugins from plugin_cache_dir. That is, export them back to a .php file, store them on a folder, and tell TXP to load them from there. Plugin ied_plugin_composer helps on this task.
try to implement a one-way only development/deployment cycle, in the sense that stuff goes from your local copy, to a staging copy, and from there, to a live copy. Of course, DB contents will get ouf of sync between live site & local/stage site. That’s something that we have yet to live with… :/
do not version content files (stuff you/end-user uploads and that TXP saves in /images/ & in /files/), or at least, don’t do it on the same repository where you are versioning your development work. This is usually an advice made by experts that I still find hard to understand & explain to others. But, believe me, don’t do it, specially if teamworking. You may regret it.
- related to above tip, I suggest you to store your background-images into /static/css/images/.

All the above suggestions doesn’t yet solve the problem with contents (and other stuff saved in DB) going out of sync between a live website and local/stage copies.
Re-syncing stuff usually needs some manual work. The only advice here is: backup your databases before stepping over them :)

maniqui · 2011-11-24 18:58:29

Ah, so you think that’s all I’ve done to complicate things up?

While you still try to digest the post above, let me tell you that I’ve also implemented a way to leverage the TXP multi-site capabilities into a development workflow.
Also, regarding the tip of running plugins from plugin_cache_dir, I also leveraged that, by having a huge folder, filled in with many TXP plugins (and, just in case I need them, their base64 installers). Then, plugins can be installed in the different TXP projects I’m working on by: (a) symlinking to them from the particular plugin_cache_dir on project, or (b) just copying them from this huge folder into the plugin_cache_dir.

The motivations behind this mind-boggling stuff:

to have just one (or, to be more precise, just a few: one for each recent TXP release) Textpattern installation, and share it among the different TXP projects I’m working on. This reduces the amount of copies of TXP that you have to “install”.
make it easier to upgrade from one TXP version to other. In this setup, upgrading TXP usually means just re-pointing one symlink.
solve some issues with the way TXP multi-site setup was designed: on a TXP multi-site setup, websites live inside a /sites/ folder on the root folder of TXP. I didn’t like that because: (a) it makes it harder to update a website to a new TXP version without affecting all the websites, (b) it isn’t teamwork & DVCS friendly, © it’s TXP multi-sites done ~~right~~ wrong :)
to “separate concerns”, as I’ve:
- one Git repository for Textpattern core files: this also allows me to easily fork TXP and have modded/hacked versions.
- one Git repository for TXP plugins (the huge folder I mentioned): this also allows me to store and shared modded/hacked versions of plugins.
- and one Git repository for each TXP project (i.e. for each personal/client website project).
How do they work all together? Via the magic of 2 or 3 symlinks placed strategically, that ties these 3 components (TXP core, plugins and particular project) together.

Also, my opinion is that this kind of setup “pays off” even for a single project. And, of course, it will “pay off” even more on subsequent TXP projects.

If there is interest, I’ll go into the detail of this setup on a series of posts (here at TXP forum). It’s not that is too complicated to explain (and understand), but, of course, it’s definitely a bit more complicated (but not that much) than a simple TXP install.

Edit: I edited to state that the way TXP currently provides multi-site capabilities is wrong (imho), or better said, it can be done better. I’ll try to provide justification to this statement on some following post.

Last edited by maniqui (2011-11-24 20:10:32)

jakob · 2011-11-25 15:58:43

jakob wrote:

Versions for SVN … Tower for Git (unfortunately each is native to its own versioning system and don’t work for both git and svn).

I should revise what I wrote previously. The makers of Tower added git-svn support a few days ago. Here’s their blog post on using it. Ross, because you are in an education environment, you might qualify for a 50% discount too…

Gocom · 2011-11-25 21:58:43

Julián, nice posts. And the way of you manage all that is pretty good too. I like it when I’ve used it. Bit confusing to set up tho.

maniqui wrote:

run plugins from plugin_cache_dir. That is, export them back to a .php file, store them on a folder, and tell TXP to load them from there. Plugin ied_plugin_composer helps on this task.

Hard drives are not fast. And sorting arrays isn’t that fast either. Instead of using plugin cache directory, I use single loader plugin (loaded to the database), which then directly includes specific files. I.e.

include rah_assets . '/plugins/cnk_versioning.php';
include rah_assets . '/plugins/rah_toolkit.php';
include rah_assets . '/plugins/rah_backup.php';
include rah_assets . '/plugins/rah_post_versions.php';

You could also chain number of files. Plugin loaded in DB includes one file that then includes the plugins. I.e.

include rah_assets . '/load/'. txpinterface .'.php';

Anyways, it offers easy way to load the code the order you want (without renaming symlinks) and avoids globbing around.

As far as the plugin template goes, you can strip those out. Textpattern doesn’t need that – the whole template. You can just copy the actual code, without all the help documentation and headers which are not used for anything other than packaging the installer file (and by ied_plugin_composer).

solve some issues with the way TXP multi-site setup was designed: on a TXP multi-site setup, websites live inside a /sites/ folder on the root folder of TXP. I didn’t like that because: (a) it makes it harded to update a website to a new TXP version without affecting all the websites, (b) it isn’t teamwork & DVCS friendly, © it’s TXP multi-sites done ~~right~~ wrong :)

Have never completely understood that way of handling multiple sites. In my opinion it’s much easier to just have single install, and then three symlinks per site (index.php, css.php and textpattern). Then in the config.php (or by other means) you just set the different database access details depending on the site (requested host, or a path). In the most simplest case config.php could include something like;

<?php
define('rah_assets', 'foo/the/assets');
switch($_SERVER['HTTP_HOST']) {
	case 'site1.example.com':
		$txpcfg['user'] = '';
		$txpcfg['pass'] = '';
		$txpcfg['host'] = 'localhost';
		$txpcfg['table_prefix'] = '';
		$txpcfg['dbcharset'] = 'utf8';
		$txpcfg['db'] = '';
		$txpcfg['txpath'] = 'foo/bar1';
	break;
	case 'site2.example.com':
		$txpcfg['user'] = '';
		$txpcfg['pass'] = '';
		$txpcfg['host'] = 'localhost';
		$txpcfg['table_prefix'] = '';
		$txpcfg['dbcharset'] = 'utf8';
		$txpcfg['db'] = '';
		$txpcfg['txpath'] = 'foo/bar2';
	break;
	case 'example2.com':
		$txpcfg['user'] = '';
		$txpcfg['pass'] = '';
		$txpcfg['host'] = 'localhost';
		$txpcfg['table_prefix'] = '';
		$txpcfg['dbcharset'] = 'utf8';
		$txpcfg['db'] = '';
		$txpcfg['txpath'] = 'foo/bar3';
	break;
	default: die();
}
?>

…and the installation can power 110 different Textpattern installations. Heck, you could even make virtualhost’s documentroot to point to the exact same directory and have no symlinks at all. And want to move admin side? Just change your wanted domain’s documentroot to textpattern or use a symlink. If symlink named admin is in /foo/bar/plob then the admin-side interface is there too.

Last edited by Gocom (2011-11-25 22:11:01)

artagesw · 2011-11-25 23:13:51

Regarding the default multi-site configuration…

This configuration (nested sites directory) was provided by default because it was the configuration that would be compatible with the highest number of web hosts. Many cheap web hosts restrict customers from accessing the server’s file system outside of the web server’s document root. In this case, the nested sites directory is the only mechanism available for multi-site.

Personally, I don’t use such restricted hosts and I have never implemented multi-site that way for my own sites. The preferred configuration is to install the Textpattern core files in a single shared location outside of the web server’s document root. Then, to use a separate virtual host directory for each separate site, each of which references the Textpattern core directory via symlinks.

Jukka’s config file trick is clever. But I prefer to have a private configuration directory associated with each site (again, outside of the document root) for config files, etc. This keeps each site independent of all others.

planeth · 2011-11-29 09:32:41

maniqui wrote:

If there is interest, I’ll go into the detail of this setup on a series of posts (here at TXP forum). It’s not that is too complicated to explain (and understand), but, of course, it’s definitely a bit more complicated (but not that much) than a simple TXP install.
Edit: I edited to state that the way TXP currently provides multi-site capabilities is wrong (imho), or better said, it can be done better. I’ll try to provide justification to this statement on some following post.

I am deeply interested in an in-depth tutorial (or what would be the most suitable).
Although i’m using cnk_versioning, I still haven’t wrapped my mind around version control stuff.

And I also find that the multi-site setup quite difficult to handle, especially if you want to use simlinks.

Destry · 2011-11-29 10:50:07

You guys — Maniqui — please put this great information in Multi-site installation. It’s absolutely buried from the world in this thread, which isn’t even titled suggestively to what you’re sharing. Purdy please?

If everyone has a different approach, no big deal… create new sections in the page that describes the different tacs. I’ll even edit the page for conciseness, organization, etc… just get the goods in there.

Ed. And notice on the wiki home page the link is prominent under “Installation”. The page can then be cross-referenced with the file tree docs, etc.

Last edited by Destry (2011-11-29 10:52:48)

Algaris · 2011-11-29 12:40:51

@Bloke. Thank you for posting your script. It’ll be very useful.
btw if you happened to buy a Mac last Friday you’d have got a hefty discount due to Apples Black Friday sale. That’s how I got mine a few years back, along with using Quidco to get a nice chunk of cash back.

maniqui wrote:

My nickname was mentioned more than 3 times, therefore, I have been summoned.

Kind of like summoning the Devil, only we don’t loose our soul ;-)

Thank you so much for taking the time to post in this thread. There’s a huge wealth of information to work though here. It’s a little overwhelming to wrap my head round it all.

I agree with pretty much everything you’ve said on the development side. Keeping all of that out of the database is a big help and cnk_versioning is a godsend in this respect (I just hope that someone will take the time to adopt and maintain it).

Now if only there was an easy way to work with content (articles, etc) as you mentioned.

maniqui wrote:

version your code, even if you are doing work in solo-mode. May this be a great opportunity to learn some modern DVCS like Git (this website will really help you if you are a bit afraid: http://gitimmersion.com/).

This is something I have been looking into after struggling to keep two branches in sync and then having to write off my main development branch and start again.

I’m not very good with the Terminal and find little joy in using it which is why I was considering using GIT Tower (http://www.git-tower.com). I’m assuming from Jakob’s posts this should be able to fit into a similar workflow to yours?

Is it possible to host my main repository on my Mac (or even a virtual Windows server that the IT department set up to host my development work)? What are the benefits of externally hosting using services like Beanstalk as opposed to hosting your own repository?

maniqui wrote:

If there is interest, I’ll go into the detail of this setup on a series of posts (here at TXP forum). It’s not that is too complicated to explain (and understand), but, of course, it’s definitely a bit more complicated (but not that much) than a simple TXP install.

If you have the time I really would be interested in learning more about your setup.

jakob · 2011-11-29 13:15:46

Kind of like summoning the Devil…

No, no, he’s a genie.

jstubbs · 2011-11-30 02:04:50

Destry wrote:

You guys — Maniqui — please put this great information in Multi-site installation. It’s absolutely buried from the world in this thread, which isn’t even titled suggestively to what you’re sharing. Purdy please?

I just came across this thread myself. Will you be sending this tutorial to TXP Tips, maniqui?

jakob · 2012-02-17 08:14:38

I’m not very good with the Terminal and find little joy in using it which is why I was considering using GIT Tower.

Ross, if you haven’t plumped for it already, it’s available at 25% off until the end of Feb -> http://www.git-tower.com.

Algaris · 2012-02-17 08:18:09

Interesting. I wonder if that 25% will stack with the educational discount. Thanks for pointing this out.

Textpattern CMS

Textpattern CMS support forum

#37 2011-11-24 17:43:17

Re: Working with live and development sites

#38 2011-11-24 18:58:29

Re: Working with live and development sites

#39 2011-11-25 15:58:43

Re: Working with live and development sites

#40 2011-11-25 21:58:43

Re: Working with live and development sites

#41 2011-11-25 23:13:51

Re: Working with live and development sites

#42 2011-11-29 09:32:41

Re: Working with live and development sites

#43 2011-11-29 10:50:07

Re: Working with live and development sites

#44 2011-11-29 12:40:51

Re: Working with live and development sites

#45 2011-11-29 13:15:46

Re: Working with live and development sites

#46 2011-11-30 02:04:50

Re: Working with live and development sites

#47 2012-02-17 08:14:38

Re: Working with live and development sites

#48 2012-02-17 08:18:09

Re: Working with live and development sites

Board footer