Performance Anxiety | custom_field loopiness

etc · 2015-03-25 12:08:37

I guess I have misunderstood your intentions, sorry. No, there will be no problem if you put all links in a form, but yes, then you delegate computers work to humans.

Actually, 2000 might be not such a big deal. Robert (wet) has written a plugin that can generate thousands of articles at once, for testing.

giz · 2015-03-25 23:28:29

Thanks, I’ll try wet_lorem_ipsum – it looks to be what I need.

For the moment I’m using your etc_query solution (so much better than my hack) – it’ll act as a baseline for my performance testing, with Jukka’s solution there as a back-up if the server squeals.

Thanks all for introducing me to new depths of Textpattern. All very useful.

etc · 2015-03-26 09:34:09

You can go with pure php, as here, it will be a little faster.

etc · 2015-03-27 18:59:28

giz wrote #289333:

Can someone explain why I couldn’t just wrap ask_cache around my inefficient code? Not that I’ve ever used it before…

I come back to your question: because some visitor has to trigger cache updates, and he/she will have to wait while the inefficient code is processed. Now, why not to update aks_cache automatically, on each article post/save? Here is my proposal:

Install aks_cache and check Reset cache if site was updated preference on aks_cache tab.
Wrap your code in some long-living aks_cache block, with noreset="".
Install and activate etc_ping (below), and fill etc_ping field in Advanced Preferences with one or several comma-separated URLs where your aks_cache block(s) is (are) visible.

Each time an article is saved/updated, etc_ping will connect to its URLs and trigger aks_cache update.

# Name: etc_ping v0.1 
# Type: Admin plugin
# 
# Author: 
# URL: 
# Recommended load order: 5

# .....................................................................
# This is a plugin for Textpattern CMS - http://textpattern.com/
# To install: textpattern > admin > plugins
# Paste the following text into the 'Install plugin' box:
# .....................................................................

YToxMTp7czo0OiJuYW1lIjtzOjg6ImV0Y19waW5nIjtzOjY6ImF1dGhvciI7czowOiIiO3M6
MTA6ImF1dGhvcl91cmkiO3M6MDoiIjtzOjc6InZlcnNpb24iO3M6MzoiMC4xIjtzOjExOiJk
ZXNjcmlwdGlvbiI7czowOiIiO3M6NDoiY29kZSI7czo3MzU6InJlZ2lzdGVyX2NhbGxiYWNr
KCdldGNfcGluZycsICdhcnRpY2xlX3Bvc3RlZCcpOw0KcmVnaXN0ZXJfY2FsbGJhY2soJ2V0
Y19waW5nJywgJ2FydGljbGVfc2F2ZWQnKTsNCnJlZ2lzdGVyX2NhbGxiYWNrKCdldGNfcGlu
Z19pbnN0YWxsJywgJ3BsdWdpbl9saWZlY3ljbGUuZXRjX3BpbmcnKTsNCg0KZnVuY3Rpb24g
ZXRjX3BpbmdfaW5zdGFsbCgkZXZlbnQ9JycsICRzdGVwPScnKQ0Kew0KCWdsb2JhbCAkcHJl
ZnM7DQoJaWYoJHN0ZXAgPT0gJ2RlbGV0ZWQnKSB7DQoJCXNhZmVfZGVsZXRlKCd0eHBfcHJl
ZnMnLCAibmFtZSBMSUtFICdldGNcX3BpbmcnIik7DQoJfQ0KCWVsc2VpZigkc3RlcCA9PSAn
ZW5hYmxlZCcpIHsNCgkJaWYoIWdldF9wcmVmKCdldGNfcGluZycpKSBzZXRfcHJlZignZXRj
X3BpbmcnLCAkcHJlZnNbJ2V0Y19waW5nJ10gPSAnJywgJ2V0Y19waW5nJywgUFJFRl9BRFZB
TkNFRCwgJ2xvbmd0ZXh0X2lucHV0Jyk7DQoJfQ0KCXJldHVybjsNCn0NCg0KZnVuY3Rpb24g
ZXRjX3BpbmcoJGV2ZW50LCAkc3RlcCwgJHJzKSB7DQoJaWYoJHVybHMgPSBnZXRfcHJlZign
ZXRjX3BpbmcnKSkgew0KCQkkdXJscyA9IGRvX2xpc3QoJHVybHMpOw0KCQlmb3JlYWNoKCR1
cmxzIGFzICR1cmwpIHsNCgkJCWlmKCFwcmVnX21hdGNoKCcvXmh0dHBzPzovaScsICR1cmwp
KSAkdXJsID0gaHUubHRyaW0oJHVybCwgJy8nKTsNCgkJCWZpbGVfZ2V0X2NvbnRlbnRzKCR1
cmwpOw0KCQl9DQoJfQ0KfSI7czo0OiJ0eXBlIjtzOjE6IjQiO3M6NToib3JkZXIiO3M6MToi
NSI7czo1OiJmbGFncyI7czoxOiIyIjtzOjQ6ImhlbHAiO2I6MDtzOjM6Im1kNSI7czozMjoi
ZDhhN2VkYjc1ZDg3NTBmYWJlMDI5ZDgxODMyNGM4YmUiO30=

Devs: would be great if update_lastmod() fired a callback.

Update: now allows relative URLs (like index.php?id=1).

Last edited by etc (2015-05-11 20:17:58)

Bloke · 2015-03-27 20:11:54

etc wrote #289510:

would be great if update_lastmod() fired a callback.

Fabulous idea. Can you please raise a feature or submit a pull request and we’ll get it folded in.

giz · 2015-03-27 20:17:29

Thank you Oleg: great stuff. I’ll give it a whirl and report back with how my performance testing goes.

etc · 2015-03-27 21:06:17

Bloke wrote #289511:

Fabulous idea. Can you please raise a feature or submit a pull request and we’ll get it folded in.

Thanks Stef, done. Would be even better if update_lastmod() were more verbose re the event triggering the update, but it makes 30 requests.

Bloke · 2015-03-27 21:23:37

etc wrote #289513:

Would be even better if update_lastmod() were more verbose re the event triggering the update

Thanks. If you have any ideas on callback name/event/step, by all means share them on the Issue page. We could raise a single one and you would have to use the $event to determine if it’s one you want to intercept, or maybe core could set the $step as the event for more granular options?

etc · 2015-03-27 21:52:17

Something like this?

function update_lastmod($event = '', $rs = null) {
	...
//	$event will be $step, $rs contains article (image, ...) data passed by the caller
	callback_event('update_lastmod', $event, false, $rs);
}

This should be granular enough, but we can sleep on it for a while.

Edit: sorry for polluting the thread, maybe move it to Feature ideas?

Last edited by etc (2015-03-27 22:12:17)

giz · 2015-04-10 21:08:37

etc wrote #289510:

Install aks_cache and check Reset cache if site was updated preference on aks_cache tab.
Wrap your code in some long-living aks_cache block, with noreset="".
Install and activate etc_ping (below), and fill etc_ping field in Advanced Preferences with one or several comma-separated URLs where your aks_cache block(s) is (are) visible.

If my aks_cache code block appears on every page, what should I enter in etc_ping’s field in Advanced Preferences?

A minor issue: etc_ping’s preferences do not appear under 4.6dev. I know how to get around this!

Thank you.

etc · 2015-04-10 21:21:50

giz wrote #289901:

If my aks_cache code block appears on every page, what should I enter in etc_ping’s field in Advanced Preferences?

Any page will do, if aks_cache blocks share the same id.

A minor issue: etc_ping’s preferences do not appear under 4.6dev. I know how to get around this!

Please share, I’m not (yet) very familiar with 4.6.

giz · 2015-04-10 22:08:57

Thanks.

etc wrote #289902:

Please share, I’m not (yet) very familiar with 4.6.

Nothing spectacular: I’ve installed etc_ping on my 4.5.7 site, and plan on copying the relevant prefs over to my 4.6dev performance-testing installation, via phpmyadmin.

.

Or did you mean “explain the bug”? In my 4.6dev install there is no ‘advanced prefs’ tab – every preference (other than etc_ping) is shown in a large list, broken down by toggle-able sections. No mention of etc_ping in the source either. I hope this helps!

Last edited by giz (2015-04-10 22:15:34)

jakob · 2015-05-11 15:49:29

etc wrote #289510:

Now, why not to update aks_cache automatically, on each article post/save?

I’ve been adding a series of aks_cache blocks to a website and it makes a real difference, especially where tag use is intensive. But like you mention, the cache only persists for as long as the cache time (default = 60 mins) and has to be regenerated from scratch at intervals although nothing has changed. It could, however, conceivably remain cached until the next page update.

From your description, etc_ping sounds like it would be ideal, but I’m not sure if I understand the concept: am I right in assuming that when you save or post an article, etc_ping causes the pages listed in the etc_ping pref to be ‘visited invisibly’, causing aks_cache to refresh those cache blocks in that moment (rather than when the next person happens to call up that page)? In other words, you’re the one doing the ‘waiting’ before someone else has to.

I presume you then need to put http://www.domain.com/apples/,http://www.domain.com/oranges/,http://www.domain.com/category/fruit,… and so on in the pref, which can get quite long if you happen to have an extensive site (or can you do just /apples,/oranges,/category/fruit,…?).

Would it be possible to somehow refresh just certain aks_cache blocks, based on the article that is being saved, e.g. in the same section, or a designated section (occasionally I use sections to input data, but display it on another section)? Or would that not work, because any change to lastmod will cause aks_cache to dump its entire cache?

Finally, could the plugin be made to hook into some other event. I’m thinking of data brought in from a non-txp source, e.g. using your own etc_query?

Also, out of interest, what would you recommend as a long refresh interval: how large is a good large value: one month of minutes? one year of minutes?

etc · 2015-05-11 17:50:29

jakob wrote #290654:

From your description, etc_ping sounds like it would be ideal, but I’m not sure if I understand the concept: am I right in assuming that when you save or post an article, etc_ping causes the pages listed in the etc_ping pref to be ‘visited invisibly’, causing aks_cache to refresh those cache blocks in that moment (rather than when the next person happens to call up that page)? In other words, you’re the one doing the ‘waiting’ before someone else has to.

Yes, absolutely, etc_ping “visits” the supplied URLs on article updates, triggering aks_cache refresh (if present on the page).

I presume you then need to put http://www.domain.com/apples/,http://www.domain.com/oranges/,http://www.domain.com/category/fruit,… and so on in the pref, which can get quite long if you happen to have an extensive site (or can you do just /apples,/oranges,/category/fruit,…?).

Still yes, but it can be easily adapted to include “local” URLs.

Would it be possible to somehow refresh just certain aks_cache blocks, based on the article that is being saved, e.g. in the same section, or a designated section (occasionally I use sections to input data, but display it on another section)? Or would that not work, because any change to lastmod will cause aks_cache to dump its entire cache?

This could be done, though you are right, the blocks not updated by etc_ping will be updated (unless they are noreset) by the first visitor. One needs to modify aks_cache to fix this.

Finally, could the plugin be made to hook into some other event. I’m thinking of data brought in from a non-txp source, e.g. using your own etc_query?

I’ve raised an issue for other txp events, but haven’t thought of “custom” updates. Updates should not be triggered by visitors, otherwise there is no point in etc_ping, but etc_query is public-side… don’t know. You can wrap “external” data in <txp:aks_cache noreset="1" /> lasting, say, 59 minutes, and subscribe the appropriate pages to some free web monitoring service with 1h interval. But their synchronization is tricky.

Also, out of interest, what would you recommend as a long refresh interval: how large is a good large value: one month of minutes? one year of minutes?

After a week, txp sends the current date as Last-Modified (bug? feature?) even if nothing was modified, ~~so more than one week seems pointless~~, but txp LastMod is not changed, so whatever you want.

Note: etc_ping will be pretty useless if you often update other things than articles, unless we subscribe it to more callbacks.

Last edited by etc (2015-05-11 20:38:05)

jakob · 2015-05-12 07:56:42

Thanks for the brilliant and informative reply! For sites that are only updated sporadically – perhaps a couple of blog articles a month etc. – then a cache update on article update would be fine, I think. The only real ‘problem’ is when making site changes, but after development that is less frequent.

I’m still unsure what the best caching strategies are, and whether txp-based or server-based is better (memcache?). I’d be interested in what people do or recommend. At present I do:

cache distinct chunks of a homepage built with txp-tags (anything that involves several db queries) using aks_cache
name chunk ids/blocks to be granular enough to cover all the different display situations (e.g. different active states on different pages), but common enough to avoid creating duplicate identical chunks (e.g. sidebar article menus per year not created for every article in that year).
not cached: straightforward txp:article output unless it brings in lots of associated data (e.g. image galleries, link lists, file downloads via custom fields).
not cached: information sourced from other sites that has its own update rhythm (e.g. twitter feed etc. – for that I’m using tweetledee for that which has its own caching).

Refresh just certain aks_cache blocks, based on the article that is being saved, e.g. in the same section, or a designated section … This could be done, though you are right, the blocks not updated by etc_ping will be updated (unless they are noreset) by the first visitor. One needs to modify aks_cache to fix this.

Agreed, aks_cache just dumps the entire cache at present.

Note: etc_ping will be pretty useless if you often update other things than articles, unless we subscribe it to more callbacks.

Do you mean things like files and images that might be automatically included in article output without needing to update the article? I guess those would need excluding from long-duration cached blocks or they would not appear.

Textpattern CMS support forum

#16 2015-03-25 12:08:37

Re: Performance Anxiety | custom_field loopiness

#17 2015-03-25 23:28:29

Re: Performance Anxiety | custom_field loopiness

#18 2015-03-26 09:34:09

Re: Performance Anxiety | custom_field loopiness

#19 2015-03-27 18:59:28

Re: Performance Anxiety | custom_field loopiness

giz wrote #289333:

#20 2015-03-27 20:11:54

Re: Performance Anxiety | custom_field loopiness

etc wrote #289510:

#21 2015-03-27 20:17:29

Re: Performance Anxiety | custom_field loopiness

#22 2015-03-27 21:06:17

Re: Performance Anxiety | custom_field loopiness

Bloke wrote #289511:

#23 2015-03-27 21:23:37

Re: Performance Anxiety | custom_field loopiness

etc wrote #289513:

#24 2015-03-27 21:52:17

Re: Performance Anxiety | custom_field loopiness

#25 2015-04-10 21:08:37

Re: Performance Anxiety | custom_field loopiness

etc wrote #289510:

#26 2015-04-10 21:21:50

Re: Performance Anxiety | custom_field loopiness

giz wrote #289901:

#27 2015-04-10 22:08:57

Re: Performance Anxiety | custom_field loopiness

etc wrote #289902:

#28 2015-05-11 15:49:29

Re: Performance Anxiety | custom_field loopiness

etc wrote #289510:

#29 2015-05-11 17:50:29

Re: Performance Anxiety | custom_field loopiness

jakob wrote #290654:

#30 2015-05-12 07:56:42

Re: Performance Anxiety | custom_field loopiness

Board footer