Future-safe the archives

mistersugar · 2015-10-03 21:39:10

Dave Winer writes again about the problem with archiving our web content.

His suggestions include this:

Work with the developers of your blogging or content management software to add automatic archiving. Make sure everything you create for the site is stored safely on a backed up server.

Is there a way to do this with Textpattern? A plugin, or set of plugins?

If there’s no current way, could we build automatic archiving into Textpattern?

I’m personally interested in this because I’m moving servers again, and thinking of strategies for moving my blogging from one Txp-powered site to another Txp-powered site in a way that preserves 10 years of posts on the one site and few years of posts on the other.

Last edited by mistersugar (2015-10-04 15:51:43)

colak · 2015-10-04 07:37:30

Unless you are a programmer who has the knowledge to fix issues as they arise, here’s my 2 cents re the title of your post:

Use as few plugins as possible. The evolution of txp as well as PHP and MySQL can not guarantee compatibility of unsupported plugins.
Back up your site via ftp but also your database and store them both in your computer and in the cloud (dropbox, copy, mega, etc)
This could be a feature request: export and write the txp articles database on html documents somewhere on the site and have the content retrieved from there. Static sites are easier to keep online should any of the technologies become obsolete or broken for whatever reason.
Follow html official versions and try to keep the site updated. Many tags become deprecated and browsers stop supporting them (<blink> and <acronym> tags come to mind).
Follow css official versions … as above.
Use as little or no JS if possible … if content is what interests you.

Regarding moving your sites check out this tip. Although it looks scary, it is very straight forward and can migrate databases larger than the limitations imposed by phpMyAdmin.

You can also use the rss_admin_db_manager plugin to back up your db from the txp interface sporadically.

Last edited by colak (2015-10-04 08:21:44)

hcgtv · 2015-10-04 12:50:57

mistersugar wrote #295369:

If there’s no current way, could we build automatic archiving into Textpattern?

This is what I use, it’s a rotating backup bash script for your databases.

AutoMySQLBackup, I’m using version 2.5.

Monday to Saturday rotating gzipped SQL dumps.
Sunday weekly backup, holds the last 5 weeks
Monthly backup, forever, it’s up to you to prune (version 3.0 holds the last 5 months)

Upload the script to your server, configure the MySQL passwords and backup directories, put it in a cron job.

Use FTP or SFTP or rsync to backup your web space to your local machine.

Affirmative, Dave. I read you.

jakob · 2015-10-04 13:32:55

mistersugar wrote #295369:

Dave Winer writes again about the problem with archiving our web content.

Is there a way to do this with Textpattern? A plugin, or set of plugins?

For backing up the site and databases on a regular basis, there are few options:

Gocom/Jukka’s rah_backup suite, for which there are also sftp and dropbox add-on plugins. They are a collection of unofficial plugins and you’ll have to run them from the plugins directly and do some work ‘under the hood’ to get it set up.
Some people use a cron job and a script to run a database dump and then rsync the files + db dump back to another location. [ As BERT just described ]
A service like myrepono.com which saves versioned backups of the entire site files and database. You can set lots of options, re frequency and number of backups, and the pricing is quite affordable. Their billing infos and reporting is detailed and transparent, and it’s saved away from your own host. It is, of course, dependent on you staying paid up, so if you should become incapacitated for six months for some reason, you could conceivably lose your backups…

For keeping HTML copies of your site (e.g. that are not dependent on the continued functionality of textpattern and php), you could periodically use a web crawler / site sucker etc. to crawl your site and save the HTML files it produces. Most such programs interlink the files correctly too so it will work offline and online. It does depend, of course, on the complexity of your site and how much you are dependent on other web-based services (e.g. if you link to scripts directly like the Jquery CDN, you might want to think of providing local fallbacks). You also lose dynamic functionality such as the search function. Essentially what you making is a point-in-time facsimile of your site than can be read on the most basic of machines.

You can also output your articles in a form of your choosing by defining another template for them, e.g. outputting your article content without the HTML around it as plain markdown files for further processing in a text editor of your choosing. Several apps exist that will export Word files from markdown source, and I think there is a basic textile to word macro somewhere out there.

I’m personally interesting in this because I’m moving servers again, and thinking of strategies for moving my blogging from one Txp-powered site to another Txp-powered site in a way that preserves 10 years of posts on the one site and few years of posts on the other.

You could use phpmyadmin and splice together the textpattern and txp_image tables (+ files + links if necessary), but you’re likely to run into numbering conflicts (e.g. duplicate article and image id numbers) that would need resolving manually. You may also need to correct custom_field usage if your custom fields are not identical on both sites. It’s not impossible, just a bit “hands-on”.

You can export to CSV format with phpmyadmin, then merge two tables, correct numbering etc. then re-import if you want to do it manually. Just do a test to check it works before getting stuck into thousands of entries…
One strategy might be to take the smaller of the two sites and bump the article ID and image ID (+files/links IDs) by a large number (e.g. 1000 or 5,000 or 10,000), then rename your images by the same pattern, then re-import. They’ll be a gap in the numbering but you should avoid collisions that way. If your articles are ordered by date on your site the articles will be interleaved even if the ID numbers aren’t.
If you’re using custom_1 to custom_x on both sites, and not for the same things, you’ll likewise need to create new columns and then create the corresponding new custom fields in txp before re-importing.

Aside from that, this sounds like just the thing a script could do better than humans …

bici · 2015-10-04 20:08:06

monthly:
i use scripts to auto-backup my MysQL databases
rsync the html files and the mysql databases to Strongspace

Last edited by bici (2015-10-04 20:08:25)

Textpattern CMS

Textpattern CMS support forum

#1 2015-10-03 21:39:10