smd_xml : extract data from XML feeds

aslsw66 · 2011-10-26 14:33:52

At least this means I’m getting better with my understanding of plugins.

I’m happy to tinker with PHP plus the index value to calculate the date, and wait until your new plugin is ready to be released in the wild.

Bloke · 2011-10-26 14:38:17

aslsw66 wrote:

I’m happy to tinker with PHP plus the index value to calculate the date, and wait until your new plugin is ready to be released in the wild.

Cool. fwiw it’s been ready for months I’ve just not got round to formally releasing it.

towndock · 2011-10-26 16:37:15

A thumbs up for both this plugin and the beta version (all seems well in my use thus far). I just first needed smd_xml for a mission this month. It quickly and elegantly parsed the custom xml feed needed.

Another great Stef plugin that makes Textpattern a more complete CMS.

kvnmcwebn · 2011-10-27 00:16:45

Steph if I want to send articles from a specific section on one site to display on another should i start with this plug in?
thanks

tye · 2011-10-27 00:17:13

I still haven’t got back to it… but will do

Don’t think I ever said thanks stef… thanks :)

Bloke · 2011-10-27 11:31:22

kvnmcwebn wrote:

if I want to send articles from a specific section on one site to display on another should i start with this plug in?

You could, yes. Install this on the destination site and set the data URL to the RSS feed of the section you want to duplicate. Then just grab the fields you want and use the container to display what you want from the feed.

all

Thanks for the thanks. I’ve been working pretty hard on this — specifically on the SOAP stuff — and it’s finally starting to work close to how I envisaged. Notwithstanding the fact that SOAP is truly awful as a spec, I’ve managed to work around a lot of the kinks and it can handle a variety of SOAP data formats. Would love people to be able to test it more before I consider it releasable so if anyone can let me know how it performs, or can post a URL to a SOAP implementation I can use to test it, that would be greatly appreciated.

I’ve also uploaded another new version of the beta this morning. This version:

has better, more complete documentation
removes the forced whitespace eradication: it’s now an optional “transform” called kill_spaces
has more format="escape" options so you can translate double and single quotes
is slightly refactored internally to remove some duplicated code
has a new timeout attribute to allow you to configure how long before the plugin gives up waiting for a feed (default: 10 seconds)

I don’t envisage any problems with the whitespace part, but I haven’t tested it much yet so any reports of its success or otherwise would be handy. Many thanks.

P.S.: If anybody finds the built-in formatting isn’t enough or you want to apply formatting to {replacements} in ontag Forms, you can also download a copy of smd_wrap from the same location as the beta smd_xml. It does a lot more besides, of course!

Last edited by Bloke (2011-10-27 12:49:57)

kvnmcwebn · 2011-10-29 23:25:49

cool

Mats · 2011-10-30 05:42:09

This feed stopped working with the new beta:

<rsp stat="ok">
<photos page="1" pages="28" perpage="20" total="555">
<photo id="6254947247" owner="8957600@N02" secret="f28b9d35b0" server="6224" farm="7" title="Vi åkte till turistmålet Eketorps Borg mitt ute på södra Ölands Alvar" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6236699194" owner="68320803@N07" secret="ce589d571f" server="6235" farm="7" title="Eketorps borg - excavated & reconstructed fortress" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6236698924" owner="68320803@N07" secret="25021fbcb6" server="6045" farm="7" title="Eketorps borg - excavated & reconstructed fortress" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6158791349" owner="42534216@N03" secret="e75d582e6f" server="6087" farm="7" title="Fortress of Eketorp II" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6128447376" owner="28299806@N07" secret="ed1b8ae290" server="6084" farm="7" title="Eketorp (Öland)" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6098026509" owner="98302910@N00" secret="e54e4a12e2" server="6189" farm="7" title="IMG_4423" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6074429622" owner="66704173@N05" secret="c601177033" server="6075" farm="7" title="Eketorp" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6072563916" owner="38538748@N03" secret="6ac17da44b" server="6071" farm="7" title="216 Eketorps borg." ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6058493645" owner="49592306@N07" secret="2c4925cba3" server="6197" farm="7" title="Eketorps borg." ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6059041956" owner="49592306@N07" secret="e823097451" server="6194" farm="7" title="Eketorps borg." ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6058493921" owner="49592306@N07" secret="8d1498941a" server="6072" farm="7" title="Eketorps borg." ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6046611553" owner="22539639@N02" secret="f839e983a0" server="6087" farm="7" title="Eketorps fornborg" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6046611647" owner="22539639@N02" secret="6d9524f962" server="6061" farm="7" title="Eketorp" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6015972212" owner="42563570@N06" secret="cc7cfa5316" server="6015" farm="7" title="Mother and Daughter at Eketorps borg - Öland, Sweden" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="6015972132" owner="42563570@N06" secret="597bc707d7" server="6136" farm="7" title="Eketorps Borg in Öland, Sweden" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="5989007166" owner="42534216@N03" secret="7ca0e4ded9" server="6137" farm="7" title="Fortress of Eketorp I" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="5975921886" owner="30550132@N06" secret="d233bc2201" server="6122" farm="7" title="SE - Agricultural Area of Southern Öland - Eketorp" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="5973086501" owner="11373239@N04" secret="350ef0e22b" server="6149" farm="7" title="Hauseingang auf Eketorp" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="5969345115" owner="11373239@N04" secret="08e52fcefb" server="6030" farm="7" title="Kurze Pause auf dem Weg nach Eketorp / Öland mit meiner schwedischen Freundin Anni" ispublic="1" isfriend="0" isfamily="0"/>
<photo id="5969881736" owner="11373239@N04" secret="6b01e37f51" server="6005" farm="7" title="Eketorp, der Innenbereich" ispublic="1" isfriend="0" isfamily="0"/>
</photos>
</rsp>

Bloke · 2011-10-30 11:05:10

Mats wrote:

This feed stopped working with the new beta

Hmmm, was it working with the previous beta or have you just jumped from v0.3 to the beta? If it’s the latter then I can understand it as things are handled very differently now; most notably the way character data is processed. In your feed you have raw ampersands so you may well need to encode them using the convert attribute to make it a valid feed. Here’s what attributes work for me:

record="photo" convert="&|&amp;" concat="0"

That gives me 20 records; one for each ‘photo’. That is, I assume, what you want?

Out of curiosity, what were your smd_xml tag attributes that now fail to render correctly in v0.40? Perhaps I inadvertently introduced a major regression so I’d like to find out what it is in case it’s something I can fix prior to release. Thanks.

Last edited by Bloke (2011-10-30 11:05:52)

Mats · 2011-10-30 12:01:00

It was working with the previous beta.

Here’s the code:

<txp:smd_xml data="http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=xxx&text=Eketorp&per_page=50&page=1&format=rest" record="photo" fields="photo|id, photo|farm,photo|server"  limit="50" kills_spaces="1">
<a href="http://www.flickr.com/photos/{photo|owner}/{photo|id}">
<img src="http://farm{photo|farm}.static.flickr.com/{photo|server}/{photo|id}_{photo|secret}_s.jpg" alt="{photo|title}" title="{photo|title}" />
</a>
</txp:smd_xml>

(I don’t know if you’re supposed to display the API key so i took it out. If you want the full link i can email it to you).

The problem i had was that the the output of the extract tags repeated it self three times. So the plug in was doing it’s job *3.

I’m using two smd_xml snippets on the same page (http://azap.se/eketorp/).

Bloke · 2011-10-30 13:43:21

Mats wrote:

It was working with the previous beta… the output of the extract tags repeated it self three times.

Yikes, well spotted. It was a problem with the new load_atts attribute: I was loading the attributes on every tag regardless if it was a tag or a record. Fixed in the most recent beta.

You’ll still need to supply convert="&|&" but you can remove the fields attribute entirely in your case, since your records are your fields and only contain attribute data. Also note that the kill_spaces attribute was an experimental variable that I removed in favour of transform="kill_spaces" So your tag should be something like this:

<txp:smd_xml data="http://..." record="photo" limit="50" convert="&|&amp;">
<a href="http://www.flickr.com/photos/{photo|owner}/{photo|id}">
<img src="http://farm{photo|farm}.static.flickr.com/{photo|server}/{photo|id}_{photo|secret}_s.jpg"
     alt="{photo|title}" title="{photo|title}" />
</a>
</txp:smd_xml>

Thanks for the report. Oh, and fwiw, it appears you don’t need the api_key in there: I could see the gallery no problem.

Last edited by Bloke (2011-10-30 13:44:12)

Mats · 2011-10-30 18:09:07

Thanks Stef!

tye · 2011-10-31 05:10:21

Stef – your “previous explanation“http://forum.textpattern.com/viewtopic.php?pid=253658#p253658 works great :)

<txp:smd_stef name=“god” />

:)

aslsw66 · 2011-11-06 16:10:35

Stef, I thought I would use smd_xml to standardise all of my feeds, rather than using another plugin for a Twitter feed.

Here’s my code:

<txp:smd_xml data="http://twitter.com/statuses/user_timeline/actrfs.xml" record="status" fields="id; text; created_at" skip="user" wraptag="ul" delim=";" limit="4" format="text|link;created_at|date|%a, %e/%m">
<li>
<a href="http://twitter.com/textpattern/statuses/{id}">
{created_at}
</a>
<br />{text}
</li>
</txp:smd_xml>

The result is here on a test page.

As you can see, there are two problems:

the date formatting is messed up. I have tested this a bit, and discovered that if I used a dash in the format attribute it works correctly eg. %e-%m. If I separate the backslash with spaces, it also works correctly eg. %e / %m. It only chokes when I remove the spaces.
the links are messed up. Looking in the source code, there are three hrefs being created for each line. I had a look in the source code, but that regular expression stuff is worse than Greek to me (I can actually read a few street signs in Greek!).

[EDIT]
I’m definitely using the latest version of the beta. I grabbed a fresh copy just now to make sure.

Last edited by aslsw66 (2011-11-06 16:12:10)

Bloke · 2011-11-06 19:24:49

aslsw66 wrote:

the links are messed up

Bizarre. For some reason the list of fields was being parsed twice, hence it was parsing the link into an anchor and then doing it again, destroying all the good work it had just done. Thanks for the report: fixed in the “lastest beta-beta-beta-…“http://stefdawson.com/sw/beta-plugin-downloads (this is getting silly!)

the date formatting is messed up

My initial thought after looking at the feed was that it was because the format used in the stream was not one of the approved compound formats that PHP understands. But when I tried it on my server I got Fri 4/11 for the first item, when yours reports Fri 4/05.

If the latest beta doesn’t fix it (i.e. it might have been a side-effect of erroneously doing the conversion twice) then my best guess is it’s a PHP bug. If you need me to investigate further then please let me know which version of PHP you’re running.

EDIT: please also note that I’ve flip-flopped back to forcing inter-tag whitespace removal by default. Thus it is no longer a transform, but a dedicated option again called kill_spaces. It is highly recommended to leave it on as I found it improves the performance of the internal XML parser, but if you are debugging it can make the feed unnecessarily difficult to read so you may elect to turn it off while testing attributes.

EDIT 2: incidentally, if anyone would like some particular facets of the plugin depicted in the Examples segment of the help then please let me know. If you have a feed in mind then that helps: I’ll extract the relevant parts and concoct an example from it. Thanks.

Last edited by Bloke (2011-11-06 19:55:26)

Textpattern CMS

Textpattern CMS support forum

#121 2011-10-26 14:33:52

Re: smd_xml : extract data from XML feeds

#122 2011-10-26 14:38:17

Re: smd_xml : extract data from XML feeds

#123 2011-10-26 16:37:15

Re: smd_xml : extract data from XML feeds

#124 2011-10-27 00:16:45

Re: smd_xml : extract data from XML feeds

#125 2011-10-27 00:17:13

Re: smd_xml : extract data from XML feeds

#126 2011-10-27 11:31:22

Re: smd_xml : extract data from XML feeds

#127 2011-10-29 23:25:49

Re: smd_xml : extract data from XML feeds

#128 2011-10-30 05:42:09

Re: smd_xml : extract data from XML feeds

#129 2011-10-30 11:05:10

Re: smd_xml : extract data from XML feeds

#130 2011-10-30 12:01:00

Re: smd_xml : extract data from XML feeds

#131 2011-10-30 13:43:21

Re: smd_xml : extract data from XML feeds

#132 2011-10-30 18:09:07

Re: smd_xml : extract data from XML feeds

#133 2011-10-31 05:10:21

Re: smd_xml : extract data from XML feeds

#134 2011-11-06 16:10:35

Re: smd_xml : extract data from XML feeds

#135 2011-11-06 19:24:49

Re: smd_xml : extract data from XML feeds

Board footer