TXP git repo best practices?

jakob · 2015-05-14 12:29:31

I’m intending to move an existing website over to a git repo as a test run for other sites that I work on on a long-term basis, and would be interested in your advice on best practices.

Ideally I want the following setup:

local dev site
staging site on own subdomain for testing / demoing to the customer – could conceivably auto-deploy.
production site – only manual deploy

Currently I am fine with preparing assets (e.g. image optimisation, minifying, sass processing etc.) locally so I don’t (presently) need gulp and co online. Not all my customers’ hosts would support that either.

1. What do you ignore or exclude from your git repo?

Currently, I have the this in my .gitignore file:

.DS_Store

# cms managed files
files/*
images/*

# config files
textpattern/config.php

# contents of cache directories
/cache/*
textpattern/plugins/help_cache/*
textpattern/tmp/*

# Keep all subfolder placeholder files
!.gitkeep
!.gitignore

Does it make sense not to track images and files? Is it preferable to gitignore these or to exclude them? And have I missed anything?

2. Any special advice on branching structure?

The plan is:

long-term dev branch with short-term feature/fix branches.
a staging branch – for the staging subdomain
a production branch – for the currently live site

and then to deploy from the respective branch to the respective domain.

3. What do you use to deploy your repo to the respective online domain?

Stef mentioned has mentioned beam as one manually installable version. Does anyone have experience of this?

There are also various online services like deployhq, dploy.io and ftploy that would feasibly work on hosts that don’t have git support or ssh access. Does anyone have experience of these and are they worth it?

I guess it’s important here to avoid overwriting CMS-managed assets like images or files and config files in the sync process.

4. Any advice on post-deploy hooks or scripts?

One idea – a short script to:

clear aks_cache blocks
update rah_flat (jukka has made it possible to define ea public callback hook URL for this purpose)
maybe also a variant of “etc_ping” to recreate cache blocks

5. How do you deal with database syncing?

I assume it’s necessary to have two databases (or perhaps another table_prefix) for the two sites so that experiments in the staging branch don’t affect the production site. Occasionally, though, it may be necessary to introduce a new custom_field or use a special class in the article body to implement a new feature. How do you merge the different content from the staging and production databases together?

6. How do you limit access to the staging domain to certain people?

One option is I guess rem_privileged and co, and creating logins for other visitors… Other ideas?

Lots of questions, I know! Any advice gratefully received – also about things I’ve not thought of yet. Perhaps this also helps other people who have been watching git and co, but have not yet dived into it…

Bloke · 2015-05-14 14:03:07

jakob wrote #290762:

What do you ignore or exclude from your git repo?

Your list pretty much covers it. It makes perfect sense to exclude images and files. Other options are:

.DS_Store?
.Spotlight-V100
.Trashes
Icon?
ehthumbs.db
Thumbs.db
*~
*.lock
*.log
*.sql
*.patch

Which excludes a few more OSX-specific hidden files, some potential Windows hidden files, temporary files from text editors, lock files, log files (e.g. error.log), stray DB backups, and patch files.

Depending on the nature of the site, you may also want to exclude other types of binary file that might get added outside of the /files directory, such as Microsoft office docs, and zip / tar files that you might use for exporting or archiving your repo.

Any special advice on branching structure?

That’s one way to set it up. Nothing wrong with one repo to track the various project states, but it’s not a very efficient workflow. Other people might tell you to use physically separate clones for that task, and there are benefits to that. For manually deploying, it’s preferable because, while it’s easy to do:

> git checkout production
> git merge master

in your repo, you can get yourself in a right pickle if the merge doesn’t go smoothly. It’s not a deal breaker because your production server isn’t going to be pointing at the repo anyway, since you’ll be checking out code left right and centre from various branches which changes the current state of all the project files.

You could save yourself a bit of work and just make a branch for staging because, if it goes sideways, you can manually correct merge conflicts with little harm done. Then when everything’s tickety boo on staging, you can pull out the big guns and rsync everything from staging over to the physically separate clone containing production code, to which your server points. That saves the pain of repeating a bunch of commit conflicts when deploying to production branch and then having to copy them out of there to the final resting place anyway.

But you need to remember that any time you do a git checkout, you’re changing the state of the files so if your staging server is pointing to the same repo as your dev, then it’ll keep changing too! Not what you want to happen. And if you’re having to checkout and merge a staging branch and then having to copy the code to a new area anyway to test it on your staging server, you might as well just have separate clones for each of the three, and checkout from one to the other. Saves work all round.

How do you limit access to the staging domain to certain people?

.htaccess / .htpasswd if it’s on its own subdomain?

The rest of the stuff I’ll leave to greater minds to reply.

EDIT: edited for clarity.

Last edited by Bloke (2015-05-14 14:14:14)

Destry · 2015-05-16 10:54:12

jakob wrote #290762:

I’m intending to move an existing website over to a git repo as a test run for other sites that I work on on a long-term basis, and would be interested in your advice on best practices.

Ideally I want the following setup:

local dev site

staging site on own subdomain for testing / demoing to the customer – could conceivably auto-deploy.

production site – only manual deploy

Back in the day this would have made a great TXP Mag article. A tutorial on doing exactly what you’re setting out to do, with explanations as to why one would do it, and justifications as to how (e.g., what Bloke just explained).

Keep good notes. You never know. ;)

sacripant · 2015-05-16 14:15:31

Back in the day this would have made a great TXP Mag article

+1000

and a article in rah_flat :).

5. How do you deal with database syncing?

Very good question, and big problem. No idea.

Versionning database configurations is not easy; I see rah_flat (not tested yet) can manage site preferencies, templates, sections with flat files, it’s very interresting.
But for a complete versionning site configuration, what solution about plugins (when install new plugin) and plugins configurations that have specifique table (smd_thumbnails, glz_custom_fields, smd_macros , bot_wtc)?
Actually I versionning .sql dump, .sql it’s text file and textpattern database is small.

philwareham · 2015-05-18 08:38:43

Jukka gave a good workflow for this, which I have tried to expand upon. This is what I use as a project root .gitignore:

# OS generated files #
######################
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
Icon?
ehthumbs.db
Thumbs.db

# Development files  #
######################
.sass-cache
/bower_components
/node_modules
/npm-debug.log
/scss-lint-report.xml
/tmp
/vendor

And then this is what I use as a .gitignore in the public (built) directory:

/files
/images
/index.php
/mockups
/templates/*/
/textpattern

Most of my build files are kept in a src directory and I then use Grunt to build the site. Templates are handled by rah_flat (and I use the excellent public hook feature from that plugin to update template files on the live site).

Bloke wrote #290772:

Which excludes a few more OSX-specific hidden files, some potential Windows hidden files, temporary files from text editors, lock files, log files (e.g. error.log), stray DB backups, and patch files.

Personally, I wouldn’t ignore the lock files as that makes sure collaborators are also locked to the identical dependencies as you are using – avoiding any potential issues between versions. If you are working solo then that is not so much of an issue I guess, but I’d keep lock files as part of the commits for good measure.

Take a look through Textpattern.com repo setup and that’ll give you an idea, I will update it soon with the latest workflow I use on my own stuff (it’s already set up quite well I think but I’ll expand it a bit more).

I do currently still build locally and FTP the files to the site, which I know is old-school, but I’m looking at methods to deploy better. I don’t really want to build directly on a server due to the number of dependencies you’d need to install (Ruby, Node.js, etc.) – even though that might be the most modern approach.

wet · 2015-05-18 09:00:28

philwareham wrote #290862:

This is what I use as a project root .gitignore:
…
/bower_components
/node_modules
/vendor

I’d rather have external dependencies under version control. Makes me feel safer as it gives me a chance of bisecting, finding the culprit, and reverting changes as a measure of last resort when an external update breaks a project.

Destry · 2015-05-18 09:25:06

To be honest, we (at CSF) should really do this with csf.community too, and I think we’ll be adding a forum compliment to the community as well, following Txp’s forum github guidance. For some, probably not a lot of work, but for me, a big learning experience walking through the github motions.

I wouldn’t mind some hands-on help on that, in fact, if someone learned is available from time to time. You’ll essentially become part of the CSF tech ensemble and get your name in lights, but you should be pretty comfortable with all this github stuff, as discussed in this thread. That’s why I need someone.

You’ll see Stef’s name in there already, but I assume as a Txp dev he’s got more important worries, and I’d assume the same for all other Txp devs, so I’m not really talking to you guys.

Maybe maniqui? I know he follows the CS stuff, and is even a member in our G+ community. Want to join the CSF web team, maniqui?

Last edited by Destry (2015-05-18 09:27:46)

philwareham · 2015-05-18 12:14:18

wet wrote #290863:

I’d rather have external dependencies under version control. Makes me feel safer as it gives me a chance of bisecting, finding the culprit, and reverting changes as a measure of last resort when an external update breaks a project.

They are under version control, in the bower.json, package.json and composer.json files respectively.

philwareham · 2015-05-18 12:18:32

Regarding deploy solutions – I’m testing out Mina and Capistrano and will let you know how I get on with each of them in due course.

jakob · 2015-05-18 13:08:08

Thanks for the sudden flurry of answers!

I’m not quite as advanced as you lot and as yet have only had brief skirmishes with bower, npm, composer and co, so I don’t have a trusted workflow set up with them yet. I currently use mixture.io (now free* but no longer being developed) to do sass preprocessing, minification, concatenation and image optimisation. I have that run in so-called “simple mode” (just preprocessing etc., no adhoc server) in my /assets/… (with css/scss/scripts/images/fonts subfolders) folder and add the mixture.json settings file to gitignore.

*(It says you can no longer download it but the download links are still in the html source, just commented out)

Regarding a deploy tool, I’ve now tried out beam and it was simpler and easier than I thought (video with the devs here). It runs locally – so doesn’t require node/ruby online – and uses ssh + rsync to do a differential upload/rename/move/delete so it keeps the overhead of a full s/ftp upload down. Plus you can make it deploy from your local or your remote git repo. It will also work with ftp/sftp if your online server doesn’t allow ssh access.

You install it once locally with a curl command and it installs in seconds to your main system. Then you cd to your local working directory in the terminal, and enter beam init which creates a blank config file. You then complete it with your respective deployment servers and a list of files to exclude from the sync. It’s as simple as this:

{
	"exclude": {
		"patterns": [
		  "/files/*",
		  "/images/*",
		  "/textpattern/config.php",
		  "/textpattern/cache/*",
		  "/textpattern/plugins/help_cache/*",
		  "/textpattern/tmp/*",
		  "/google*.html",
		  "/update_after_import*.*",
		  "/phpinfo.*"
		]
	},
	"servers": {
		"staging": {
			"user": "username",
			"host": "ssh-hostname.com",
			"webroot": "/path/to/staging/deploy/directory/"
		},
		"live": {
			"user": "username",
			"host": "ssh-hostname.com",
			"webroot": "/path/to/live/deploy/directory/",
			"branch": "remotes/origin/master"
		}
	}
}

replacing username with your ssh user and host with your ssh hostname/domain. The branch entry means you can lock a deploy server to a particular branch, so you can’t accidentally deploy the wrong branch to that server (I’ve yet to test that ;-).

To deploy, you:
1. cd in the terminal to your local working directory.
2. Do beam up {server name} (e.g. “beam up staging” for the staging server, “beam up live” for the live server, or whatever you choose)
3. It does a diff and shows you in the terminal window what it’s going to do, before it does anything (you can do just this as a --dry-run too). It’s just a list of files to be added, moved, deleted that you can check through.
4. It then asks you for permission (twice if it involves deleting files on the server) and ssh pass (if you’re not using a key), then does an rsync operation. It happened remarkably quickly.

The exclude patterns exclude those files and directories from the rsync, so your files, images etc. aren’t removed as part of the rsync procedure. Add to that if you have any other directories doing special things.

I’ve only had a few days experience with it, so I can’t say how it performs in the long term. Likewise, I have yet to examine any pre- and post-deploy commands, but you can specify them to be run them both locally and remotely, so you could conceivably set your site to maintenance mode to start with, then after deploying invoke the rah_flat public hook, bust caches, remove maintenance mode and etc_ping the site to regenerate your cache files on your server.

jakob · 2015-05-18 13:47:34

Phil wrote:

Jukka gave a good workflow for this

Can we see this anywhere? I thought Jukka would have found answers to a lot of these questions.

Phil wrote:

And then this is what I use as a .gitignore in the public (built) directory:

I didn’t get that bit. Is the public directory then also your remote repo? Is there a particular advantage in excluding files and images on the remote / public side as opposed to in the local repo? Do you manually add the .gitignore file to the remote repo via s/ftp?

The above deploy situation deploys from a local/remote repo to a public directory, so the remote repo and deploy directories are separate. The exclude pattern prevents files/images/cache files/google webmaster files etc. from being deleted in a sync.

Sacripant wrote:

Actually I versioning .sql dump, .sql it’s text file and textpattern database is small.

A diff comparison of two sql dumps does work if your changes aren’t numerous and your public site is not constantly being updated. You can make this a bit easier by doing an sql export from the live and staging/local database with modified settings. In phpmyadmin (or via the mysqldump command), you can:

Make it export one INSERT statement per db record (see here for an example), and omit some of the verbose table infos and CREATE infos. Make the settings the same on both database exports.
Omit certain unneeded tables from the export, e.g. aks_cache, txp_css, txp_form, txp_lang, txp_log, txp_page, txp_discuss_nonce … (adjust as needed per your database – I’ve excluded txp_css, _page, _form here because those are all updated by rah_flat, but if you use the txp css panel, you’ll need to include it.

You can then do a diff with your favourite diff tool and the entries that have changed will be highlighted. You’ll also have INSERT statements that you can copy (more or less) wholesale back into phpmyadmin or Sequel Pro, or whatever you use to administer your databases.

Sacripant wrote:

Versioning database configurations is not easy; I see rah_flat (not tested yet) can manage site preferencies, templates, sections with flat files, it’s very interesting. But for a complete versionning site configuration, what solution about plugins (when install new plugin) …?

You can get part way there with a combination of rah_flat and a plugin directory with ied_plugin_composer.

Forms / pages / css: rah_flat is easily set up and is more robust and reliable than cnk_versioning. See the instructions on Jukka’s github repo (download from releases). You set the directory with your templates in the prefs and create subdirectories for /pages, /forms, and /css. If you leave one out, it won’t be synced.

Beware that rah_flat syncs only in one direction: from files to db. It doesn’t export them first, so make a db backup or copy out the forms you need into files. It will replace your forms wholesale! It follows the same naming convention as cnk_versioning, e.g. /forms/form-name.(misc/article/link/image).txp and /pages/page-name.txp.

In debugging / testing mode it imports the templates on every page load, so don’t be worried, when you’re db-query count is suddenly very high. Once you switch to live, that only happens if you specifically call the public hook url (which you can define to be non-guessable). Despite the regular importing, I haven’t noticed a performance hit when in debug/live mode.

There’s only one real conflict I’ve noticed and that’s with adi_variables. The pane is no longer accessible and you can only continue using adi_variables by manually editing the adi_variables.misc.txp file. Jukka didn’t entertain the idea of having an exclude list, arguing that adi_variables should really employ a different method… If you switch off rah_flat when in live mode and allow your users to edit adi_variables, you must remember to make a note of any changes before switching to debug mode…

Plugins: Set a plugin directory in the txp prefs and use ied_plugin_composer to make php files of your plugins that you can then drop into the plugin directory. You can then sync or version these files. There are occasionally disadvantages with using this method (originally for developing plugins) because copying the files to the plugin directory, doesn’t necessarily trigger the install routines. ied_plugin_composer does allow you to trigger install / lifecycle events but there are times when a plugin doesn’t play well. In those cases I install using the regular method, or I install and activate regularly, then switch to the php file.

Jukka has a separate plugin for this (rah_blobin) which is supposed to overcome some of these limitations, but also has a different plugin structure. I have still to investigate this.

philwareham · 2015-05-18 14:09:00

jakob wrote #290872:

Can we see this anywhere? I thought Jukka would have found answers to a lot of these questions.

I didn’t get that bit. Is the public directory then also your remote repo? Is there a particular advantage in excluding files and images on the remote / public side as opposed to in the local repo? Do you manually add the .gitignore file to the remote repo via s/ftp?

public is the build directory, regardless of whether it is on local or a remote server. It’s where all your files are built to by the Grunt script. Effectively that would be the root folder of your website. You will need to ignore certain files in that as you won’t want them to appear in your repository. Things like Textpattern itself (which you can install via Composer), there is no need to include that in your repo, and content images/files.

Don’t confuse this with a backup snapshot of your site – they are not the same thing. The repo should contain only the development files themselves.

Jukka made the workflow in the Textpattern.com repo (well, the TXP Magazine and forum repos actually which I copied over to .com repo). Pick it apart and you should be able to understand how it works.

Beware that rah_flat syncs only in one direction: from files to db. It doesn’t export them first, so make a db backup or copy out the forms you need into files. It will replace your forms wholesale! It follows the same naming convention as cnk_versioning, e.g. /forms/form-name.(misc/article/link/image).txp and /pages/page-name.txp.

Since you are version controlling anyway, you’ll need flat files of your templates. But yes, copy all your forms/pages out of Textpattern into actual files manually if necessary when first setting up, in the directory structure expected by rah_flat.

Textpattern CMS

Textpattern CMS support forum

#1 2015-05-14 12:29:31