smd_lately: recently viewed site articles

sereal · 2010-04-28 09:54:30

hmm I see

let me try that and I’ll report it in the next 2 months :D

thanks mr.Stef

Bloke · 2010-05-10 15:35:29

Slight pimpage in the latest version 0.21:

added from, to and within so you can limit your log files searching to date & time ranges (thanks the_ghost)
added wildcard like matches to include and exclude which (I believe) are slightly faster than REGEXPs. Only use regex if you really need it for regular expression matches. In most cases you can get away with something like exclude="page:like:http, page:like:../" to filter out a whole slew of inane script-kiddie attacks on your sites
replaced a load of internal REGEXP matches with LIKE for performance reasons. Shaves a few milliseconds off the plugin, and it all counts

Give that a whirl and see how she handles.

maniqui · 2010-05-27 22:06:22

Hi Bloke,
I’m having some issues while playing with this plugin. Not sure if it’s my fault, or some plugin quirkiness, or even both.

First question first:

1) Is it compatible with TXP 4.0.7? I see you made it compatible with TXP 4.0.8, but as I’m currently doing some updates on an already hard-modded TXP 4.0.7, I’m restraining of trying to upgrade to 4.0.8 or even 4.2.0.

Now, this are the quirky quirks I’m having

minor one: when using wraptag="ul" but not specifying class attribute, I get <ul class="__FUNCTION__"> in the output.
not-so-minor-one: even if it seems to be iterating over something, I’m just getting empty output.

The code:

<txp:smd_lately by="SMD_ALL" sort="popularity asc" limit="10" wraptag="ul" break="li">
  <txp:title />
</txp:smd_lately>

That will output 10 empty <li>s.

The other weird thing is that the first time I dropped <txp:smd_lately /> tag on the template (without using as container, but in its self-closing mode, and without specifying any form), I’m pretty sure it generated some output pointing to, at least, one article. This output was wrong too, as I was getting 9 empty <li>s.

So, not sure what could be going on here, really. As always, I’ll appreciate your guidance.

Bloke · 2010-05-27 22:52:52

maniqui wrote:

Is it compatible with TXP 4.0.7?

Don’t see why not. Can’t think of anything introduced in 4.0.8+ that the plugin needs.

<ul class="__FUNCTION__">

D’oh that’s me being a tool. I put the default item in quotes when it shouldn’t be. Sorry about that. Find line 26 and remove the single quotes from around __FUNCTION__. I’ll fix that tomorrow.

I’m just getting empty output.

A few things, in no particular order:

Are you logging all hits, and do your visitor logs show entries?
What happens if you sort by descending popularity instead of showing the 10 least popular?
Are you using any complicated gbp_permanent_links magic anywhere that might be interfering with the logging?
What do you see with debug="2" (or 3) switched on? With level 3 on you should see the records for all 10 articles. Does that give you any clues why it’s misbehaving?

Past that I can’t think of anything off the top of my head. The above might trigger some further thoughts if you post your findings. I hope!

maniqui · 2010-05-28 02:15:27

Hi Bloke, thanks for your prompt reply.

<ul class="__FUNCTION__">
D’oh that’s me being a tool. I put the default item in quotes when it shouldn’t be. Sorry about that. Find line 26 and remove the single quotes from around __FUNCTION__. I’ll fix that tomorrow.

Done, and it works. Thanks.

Are you logging all hits, and do your visitor logs show entries?

Yes, I forgot to mention that, but yes, it is login correctly.

What happens if you sort by descending popularity instead of showing the 10 least popular?

No change, but don’t worry… because…

Are you using any complicated gbp_permanent_links magic anywhere that might be interfering with the logging?

We have a winner!

I also was suspecting of gbp_permanent_links, but, at a first glance, gbp_p_l wasn’t interfering with the logging. Hits were being logged successfully, so there was nothing suspicious there. I also sneak-peeked at smd_lately plugin code and see that “url_title” was a key element on the fetching of logs, so definitely there was a clue on that path…

So, after reading your question, I temporarely disabled gbp_p_l but nothing changed: smd_lately iterated over something, as in the output I could see that it was a DOM element for each iteration.
I know it was iterating over logs, because after re-initiating my tests, I cleaned/emptied all the logs, and hit a few pages, just to log a few (three or four) new entries.
And the quantity of DOM elements coincided with the quantity of articles logged on the visitors log.

So… as gbp_p_l still didn’t seem to have any relation with the issue, I continued with your next question…

What do you see with debug="2" (or 3) switched on? With level 3 on you should see the records for all 10 articles. Does that give you any clues why it’s misbehaving?

So, then, there is a debug attribute? No mention about it on plugin’s official page ;)
I should have discovered it by looking at plugin’s code… but well, I can’t see the whole Matrix yet.

So, I turned debugging, first on 2, then on 3, and looking at the output, I noticed that, as you said, it was outputting some records. But the weird thing was that it every record in the output was the same, and to be more precise, the record (the article) wasn’t any of the logged articles.

I inspected more carefully the output and I saw this line:

select *, unix_timestamp(Posted) as uPosted, unix_timestamp(Expires) as uExpires, unix_timestamp(LastMod) as uLastMod from textpattern where url_title='' AND Posted < CURDATE()

So, url=title='' was querying for articles without an url_title. I thought I had a winner there, and found that, among the +700 articles there were two without title nor url_title.
Fixed that, but no luck either. Of course, the plugin was still querying for where url_title='', but at least, this time, there were no records.

And that’s when I looked again at the plugin code, and found this comment:

// Note the regexp is anchored to the end with a $

And that’s when I heard the “click” that anyone loves to hear when tracking down this kind of issues.
I remembered (and confirmed by looking at plugin’s debug output) that gbp_permanent_links has an option, enabled by default, to append a trailing slash to permanent links.
So, logged permanent links, instead of being section/article-title, they were section/article-title/. And that’s the reason why smd_lately couldn’t process them properly and was trying to query where url_title=''.

I temporarily changed the option on gbp_p_l, logged a few new hits, and tada!! smd_lately began to output article’s content (title, etc) on each iteration. :D

Past that I can’t think of anything off the top of my head. The above might trigger some further thoughts if you post your findings. I hope!

Yes, it did trigger some further thoughts, thank you very much!

Now, well, I’ve been looking thru smd_lately code, but I couldn’t find the anchored-to-$ regexp that captures the url_title.
Do you think it could be possible to modify smd_lately to ignore any trailing slash on url_title, without affecting other trailing slash, like the ones at /section-name/?
In other words, to make it compatible with both types of url_titles: with and without trailing slash, so it will make it compatible with plugins like gbp_p_l.
In fact, vainilla TXP article URLs seems to work both with or without the trailing slash, so I would believe that TXP logs them as they are hit (with or without the trailing slash).

maniqui · 2010-05-28 03:48:18

OK, I put my programmer-wannabe hat on, and found a way to let smd_lately work with both type of URLs (with and w/o trailing slash):

Replace this line (269).

$urlparts = explode('/', $row['page']);

with this one:

$urlparts = explode('/', rtrim($row['page'],"/"));

And that’s all!

Of course, I hope Bloke blesses this change (or improves over it).

Bloke · 2010-05-28 09:07:11

maniqui wrote:

I put my programmer-wannabe hat on, and found a way to let smd_lately work with both type of URLs

Genius find, thanks maniqui. Implemented in v0.22, along with a fix for my daft class oversight.

P.S. If in doubt, there’s almost always an undocumented debug attribute in my plugins :-)

Last edited by Bloke (2010-05-28 09:14:35)

pieman · 2010-06-10 14:21:02

A client would like to show the most popular articles in the last 30 days out of those published in the last 30 days.

I set within="30 days" to limit log range, but can’t figure out how to define the date-posted limits using from.

In the smd_lately help:

from : date / time stamp (written in English) of the earliest date to consider in the logs. You may specify ?month and/or ?year to have the plugin replace it with the current month/year. Default: unset

What are the valid values we can enter here? Is there a way to do what I’m looking for without hackery?

maniqui · 2010-06-10 19:05:44

Hi pieman,

Currently, it doesn’t seem to be possible, using smd_lately, to filter the list of most popular articles, just including those published on the last 30 days.
Thinking a little about it, I think this is the correct reasoning:

1) get a list of IDs for articles published on last 30 days.
2) Sort them by popularity, based on TXP log.

I would try this approach:

1) generate a comma-separated list of IDs for articles published on the last 30 days, using smd_query or article_custom (probably not doable with article_custom)
2) generate a comma-separated list of IDs for most popular articles in the last 30 days, using smd_lately. Use a high “limit” value, so your comma-separated list of IDs is a “pool” of article ids already sorted by popularity.
3) use article_custom, and feed its “id” attribute with the list generated in 1, and the “sort” attribute, with the list generated in 2.

Like this:

<txp:article_custom id="list_of_ids_for_articles_published_in_last_30_days" sort="FIELD(ID,already_sorted_list_of_ids_for_most_popular_articles_in_last_30_days)" />

I’m not sure if the sort="FIELD(ID,...)" trick will work if you feed it with a list of IDs larger than the ones being feed to id attribute.
In that case, you may have to do some pre-processing of the already_sorted_list_of_ids_for_most_popular_articles_in_last_30_days, to remove any id that is not in the list_of_ids_for_articles_published_in_last_30_days, but at the same time, keeping the sorting.

So, if you want to test, first, I would suggest you to manually try if something like this works:

<txp:article_custom id="5,7,13,20,22" sort="FIELD(ID,54,20,7,80,13,22,33,5)" />

That is, manually pick up a list of articles, and manually/randomly sort them.
Check if the output is correct. Following the example, the articles should be output in this order: 20,7,13,22,5.

You may also like to test what happens if some article id from the last published articles, doesn’t appear in the list of most popular articles. That’s a possible scenario, if one of articles published on the last 30 days isn’t popular at all.
In other words, and similar to the test above, you may want to try something like this:

<txp:article_custom id="5,7,13,20,22" sort="FIELD(ID,54,20,7,80,13,33)" />

As you can see, the sort="FIELD(ID,...)" doesn’t include some of the ids (5, 22) that are included in the id attribute.
No idea what would happen on that case.

Well, this are some ideas off the top of my head. Hope they make some sense!

maniqui · 2010-06-26 15:14:00

Hi Bloke,

A client’s site has been deactivated by his hosting company. The hosting support informed the client that they have deactivated the website because it was taking up too many database resources.
They also informed that the offending query is this:

Here are the mysql logs traced on the server :-

Database User: dummy_user for database dummy_txp 
=================================================== 
| 39992 | dummy_user | localhost | dummy_txp | Query | 117 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 39995 | dummy_user | localhost | dummy_txp | Query | 117 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40008 | dummy_user | localhost | dummy_txp | Query | 117 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40009 | dummy_user | localhost | dummy_txp | Query | 117 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40011 | dummy_user | localhost | dummy_txp | Query | 117 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40014 | dummy_user | localhost | dummy_txp | Query | 104 | Copying to tmp table | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40015 | dummy_user | localhost | dummy_txp | Query | 93 | Copying to tmp table | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40018 | dummy_user | localhost | dummy_txp | Query | 107 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40047 | dummy_user | localhost | dummy_txp | Query | 107 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40049 | dummy_user | localhost | dummy_txp | Query | 100 | Copying to tmp table | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40050 | dummy_user | localhost | dummy_txp | Query | 17 | closing tables | select *, unix_timestamp(Posted) as uPosted, unix_timestamp(Expires) as uExpires, unix_timestamp(Las | 
| 40053 | dummy_user | localhost | dummy_txp | Query | 100 | Copying to tmp table | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40065 | dummy_user | localhost | dummy_txp | Query | 107 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40069 | dummy_user | localhost | dummy_txp | Query | 100 | Copying to tmp table | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40072 | dummy_user | localhost | dummy_txp | Query | 100 | Copying to tmp table | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40082 | dummy_user | localhost | dummy_txp | Query | 107 | closing tables | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
| 40089 | dummy_user | localhost | dummy_txp | Query | 100 | Copying to tmp table | SELECT count(page) as popularity, page, MAX(time) as time FROM txp_log WHERE 1=1 AND page NOT LIKE ' | 
===================================================

The plugin usage is the following:

<txp:smd_lately by="" sort="popularity" within="30 days" limit="10" wraptag="ul" break="li">
  <h3><txp:permlink><txp:title /></txp:permlink></h3>
  <p><txp:rvm_substr length="100"><txp:etz_striptags><txp:excerpt /></txp:etz_striptags></txp:rvm_substr></p>
</txp:smd_lately>

Do you have any idea of what could be going on with this and if there is any measure that could be taken to avoid this issue?

Thank you.

Bloke · 2010-06-27 08:41:00

maniqui wrote:

Do you have any idea of what could be going on with this and if there is any measure that could be taken to avoid this issue?

Hmmm, short answer: not sure. Granted the smd_lately tag can be a little heavy because of all the regex stuff going on — it has to exclude a lot of stuff like any potential landing pages, plus it uses 1 query for each result to grab the corresponding article info to allow you to use article tags.

In this case you only use 10 article queries and have 30 days’ worth of results to consider. To dig further we’d have to look at a few more bits of info:

How big the log table is — total number of entries — and roughly how many of those fall in the last 30 days. Consider reducing the size of the log window if possible
How often the smd_lately tag is called — is it every page or only on certain pages? Again, thinning out the density of calls might help
Your tag trace — or at least the query execution time and memory usage info. That might give us a clue of the load on certain pages
Consider wrapping the smd_latey tag in rah_runtime to find out if the lion’s share of total page execution time is actually that tag or something else is contributing
PHP/MySQL version — just for the sake of completeness

One way to circumvent this, which I could perhaps look at implementing, is a cache. If you could somehow control the freshness of the cache using some timeout option — say, 5 minutes — then you could reduce server load at the expense of less up-to-date information and balance it as you see fit. Because of the potentially large nature of the result set I’d probably cache it in a file in the tmp directory instead of the database.

In the meantime you could implement a poor-man’s cache yourself. Untested but something like this in pseudocode:

if ((current time - qui_stored_time ) > 300)
do:
   <txp:smd_lately ... /> : store result in prefs as qui_lately_tag
   Store time() in prefs as qui_stored_time
else: Retrieve qui_lately_tag

Just set qui_stored_time manually with the current time before you start. In fact I’ve just had a thought that perhaps the smd_prefalizer plugin could have a <txp:smd_set_pref> and <txp:smd_get_pref> public-side tags which would facilitate setting this up very easily. I might just do that if it’d help you out (with suitable controls to disallow clobbering TXP’s built-in prefs!) Let me know if this’d be useful.

Alternatively, if you didn’t want to mess about with the prefs directly, you could use smd_vars to stash the stored_time.

Last edited by Bloke (2010-06-27 08:41:44)

maniqui · 2010-06-27 15:07:12

Hi Bloke,
many thanks for the thoroughly reply. I’ll investigate the options, probably implementing a cache-based solution (aks_cache) as a quick solution.
BTW, Is there a release of smd_vars?

Bloke · 2010-06-27 15:10:56

maniqui wrote:

Is there a release of smd_vars?

Not officially. Could do with one really. One fine day… but hold fire for an hour or so: I’ve updated smd_prefalizer to insert/query prefs from the public side so you can be my guinea pig for that. I’ll drop you a mail once it’s documented.

jelle · 2010-09-09 12:37:03

Hi stef,

Sounds like a great plugin but I just can’t get it to work. Tried regular and zipped version, tried different tags and setting…it just won’t output anything for me.

smd_lately 0.22
txp 4.2
php 5.2

Bloke · 2010-09-09 12:39:18

jelle wrote:

Sounds like a great plugin but I just can’t get it to work. Tried regular and zipped version, tried different tags and setting…it just won’t output anything for me.

Can you please post the tag you’re using? And check you have logging set to All Results and that there are some entries in your Visitor Logs.

Last edited by Bloke (2010-09-09 12:39:41)

Textpattern CMS

Textpattern CMS support forum

#31 2010-04-28 09:54:30

Re: smd_lately: recently viewed site articles

#32 2010-05-10 15:35:29

Re: smd_lately: recently viewed site articles

#33 2010-05-27 22:06:22

Re: smd_lately: recently viewed site articles

#34 2010-05-27 22:52:52

Re: smd_lately: recently viewed site articles

#35 2010-05-28 02:15:27

Re: smd_lately: recently viewed site articles

#36 2010-05-28 03:48:18

Re: smd_lately: recently viewed site articles

#37 2010-05-28 09:07:11

Re: smd_lately: recently viewed site articles

#38 2010-06-10 14:21:02

Re: smd_lately: recently viewed site articles

#39 2010-06-10 19:05:44

Re: smd_lately: recently viewed site articles

#40 2010-06-26 15:14:00

Re: smd_lately: recently viewed site articles

#41 2010-06-27 08:41:00

Re: smd_lately: recently viewed site articles

#42 2010-06-27 15:07:12

Re: smd_lately: recently viewed site articles

#43 2010-06-27 15:10:56

Re: smd_lately: recently viewed site articles

#44 2010-09-09 12:37:03

Re: smd_lately: recently viewed site articles

#45 2010-09-09 12:39:18

Re: smd_lately: recently viewed site articles

Board footer