smd_xml : extract data from XML feeds

Bloke · 2010-02-22 09:30:34

pieman wrote:

I need to convert the timestamp into a valid format

Dammit I missed that, thanks for spotting it. I was thinking of offering a timestamp format attribute but it actually makes sense to modify the date format to handle timestamps. Try this for now until I get round to rolling it into the next release. Around line 491 where the date case is handled you’ll see this line:

$nd = strtotime($this->xmldata['{'.$sfield.'}']);

Replace it with this:

if (is_numeric($this->xmldata['{'.$sfield.'}'])) {
	$nd = $this->xmldata['{'.$sfield.'}'];
} else {
	$nd = strtotime($this->xmldata['{'.$sfield.'}']);
}

EDIT: or more succinctly use this one line:

$nd = (is_numeric($this->xmldata['{'.$sfield.'}'])) ? $this->xmldata['{'.$sfield.'}'] : strtotime($this->xmldata['{'.$sfield.'}']);

That should get you where you wanna go.

should it say {name|id} : wile_e_coyote?

Oops, typo. Will be fixed.

Last edited by Bloke (2010-02-22 09:33:08)

pieman · 2010-02-22 23:01:29

Bloke wrote:

EDIT: or more succinctly use this one line:
$nd = (is_numeric($this->xmldata['{'.$sfield.'}'])) ? $this->xmldata['{'.$sfield.'}'] : strtotime($this->xmldata['{'.$sfield.'}']);

It only works, dunnit! Thanks Bloke.

lazlo · 2010-03-01 20:38:29

Hi Stef (or others)

I have multiple non-unique fields in my xml feed and I currently when I grab them they all get merged into one field.
All the individual <MeasureTypeCode>Foo</MeasureTypeCode> tags are being joined into custom field MeasurementType { 01 02 03 08 }
All the individual <Measurement>Foo</Measurement> tags are being joined into custom field Measurement { 8.5 5.5 .607 8.5 } and so on

instead of

<MeasureTypeCode>01</MeasureTypeCode> <Measurement>8.5</Measurement> <MeasureUnitCode>in</MeasureUnitCode> being joined into Height { 8.5 in }.
<MeasureTypeCode>02</MeasureTypeCode> <Measurement>5.5</Measurement> <MeasureUnitCode>in</MeasureUnitCode> being joined into Width { 5.5 in }.
<MeasureTypeCode>03</MeasureTypeCode> <Measurement>.607 in</Measurement> <MeasureUnitCode>in</MeasureUnitCode> being joined into Depth { .607 in }.
<MeasureTypeCode>08</MeasureTypeCode> <Measurement>382</Measurement> <MeasureUnitCode>in</MeasureUnitCode> being joined into Weight { 8.5 gr }.

Sample xml

<Product>
    <RecordReference>9780889221482</RecordReference>
    <Measure>
        <MeasureTypeCode>01</MeasureTypeCode>
        <Measurement>8.5</Measurement>
        <MeasureUnitCode>in</MeasureUnitCode>
    </Measure>
     <Measure>
         <MeasureTypeCode>02</MeasureTypeCode>
         <Measurement>5.5</Measurement>
         <MeasureUnitCode>in</MeasureUnitCode>
      </Measure>
      <Measure>
         <MeasureTypeCode>03</MeasureTypeCode>
         <Measurement>0.607</Measurement>
         <MeasureUnitCode>in</MeasureUnitCode>
      </Measure>
      <Measure>
          <MeasureTypeCode>08</MeasureTypeCode>
          <Measurement>382</Measurement>
          <MeasureUnitCode>gr</MeasureUnitCode>
       </Measure>
  </Product>

My current code just makes no account for multiple <measure > tags because I am not sure how to do all under the same <RecordReference>9780889221482</RecordReference>.
I can correctly join one <measure> group under one record but not multiple <measure> groups under one record.

Any enlightenment would help.

AND if you know of a way to using the XML doctype to label custom fields that would be really helpful as well.
<!DOCTYPE ONIXMessage SYSTEM “http://www.editeur.org/onix/2.1/02/reference/onix-international.dtd”>
<MeasureTypeCode>01</MeasureTypeCode> = Height is there a way just to look this up?

regards
Les Smith

Last edited by lazlo (2010-03-01 20:50:31)

Bloke · 2010-03-03 23:20:20

lazlo wrote:

I have multiple non-unique fields in my xml feed and I currently when I grab them they all get merged into one field.

Right, this looks like it’s going to take some trickery. Assuming that your sample XML is one Record (i.e. there are multiple ‘Product’ records in your data feed) you can proceed using some txp:variable magic, and some smd_each / smd_if goodness to stitch it all together.

What I’ve done is make up two fake ‘records’ to show how it works with a data feed. I’ve assigned the records to a txp:variable for now; you may employ the direct feed in your smd_xml’s data attribute instead, though notice I’ve added the ‘fake’ container tag around the whole XML feed? For some reason the plugin didn’t like it when ‘Product’ was the top level and it only showed the first record — that might be a bug in the plugin or I might have been a bit dim when I tried it; I’ll have to track this down and see which it is!

So here’s my sample XML, which is a lot like yours but with two records in it:

<txp:variable name="the_xml">
<Fake>
  <Product>
    <RecordReference>9780889221482</RecordReference>
    <Measure>
        <MeasureTypeCode>01</MeasureTypeCode>
        <Measurement>8.5</Measurement>
        <MeasureUnitCode>in</MeasureUnitCode>
    </Measure>
     <Measure>
         <MeasureTypeCode>02</MeasureTypeCode>
         <Measurement>5.5</Measurement>
         <MeasureUnitCode>in</MeasureUnitCode>
      </Measure>
      <Measure>
         <MeasureTypeCode>03</MeasureTypeCode>
         <Measurement>0.607</Measurement>
         <MeasureUnitCode>in</MeasureUnitCode>
      </Measure>
      <Measure>
          <MeasureTypeCode>08</MeasureTypeCode>
          <Measurement>382</Measurement>
          <MeasureUnitCode>gr</MeasureUnitCode>
       </Measure>
  </Product>
  <Product>
    <RecordReference>9221482978088</RecordReference>
    <Measure>
        <MeasureTypeCode>01</MeasureTypeCode>
        <Measurement>35</Measurement>
        <MeasureUnitCode>cm</MeasureUnitCode>
    </Measure>
     <Measure>
         <MeasureTypeCode>02</MeasureTypeCode>
         <Measurement>809</Measurement>
         <MeasureUnitCode>mm</MeasureUnitCode>
      </Measure>
      <Measure>
         <MeasureTypeCode>03</MeasureTypeCode>
         <Measurement>6</Measurement>
         <MeasureUnitCode>cm</MeasureUnitCode>
      </Measure>
      <Measure>
          <MeasureTypeCode>08</MeasureTypeCode>
          <Measurement>38</Measurement>
          <MeasureUnitCode>kg</MeasureUnitCode>
       </Measure>
  </Product>
</Fake>
</txp:variable>

Before we begin this journey, we need to initialise 4 txp:variables — one for each ‘type’ with their relevant starting strings, e.g. Height { and Width {, etc. Note that the Record numbers match the {MeasureTypeCode} numbers. What we’re going to do is concatenate the relevant entries from the {Measurement} and {MeasureUnitCode} strings onto the correct txp:variable to build up a final set of variables containing the full strings.

Now comes the awkward bit. We’re going to let smd_xml pull each record out and concatenate the contents, but we’re going to use concat_delim to delimit them with a pipe. Then, inside the smd_xml container, use smd_each to iterate over the {MeasureTypeCode} list. So for each record, this will walk over the type codes in turn: 01, then 02, then 03, then 08. These 4 values have an automatic counter assigned to them in the smd_each plugin so we know where we are in the list; we’ll test this counter later with smd_if.

Inisde the main smd_each container are two further smd_each tags — both have different var_prefixes so we can distingiush the values from the outer smd_each. Essentially, they loop over each Measurement, then each MeasureUnitCode and — when they find the entry that matches the current counter — they tack the value onto the current txp:variable.

Thus, when we’re looking at the first MeasureTypeCode, the smd_if fires when we are looking at the first Measurement and then fires when we reach the first MeasureUnitCode. On the next iteration (2nd MeasureTypeCode) the smd_if fires when we reach the 2nd Measurement and the 2nd MeasureUnitCode, and so on.

The variables are built up piece by piece until, when all the smd_each tags are done, we have a complete string. These are then output before the end of the smd_xml tag so you can see them, but of course you can do what you like with them at that point.

Here’s the code anyway… hope it makes some kind of sense:

<txp:smd_xml data='<txp:variable name="the_xml" />' record="Product"
     fields="RecordReference, MeasureTypeCode, Measurement, MeasureUnitCode"
     concat_delim="|" wraptag="ul" break="li">
{RecordReference}
<br />

<!-- Define the initial state of the txp:variables; one for each MeasurementTypeCode -->
<txp:variable name="Record01">Height {</txp:variable>
<txp:variable name="Record02">Width {</txp:variable>
<txp:variable name="Record03">Depth {</txp:variable>
<txp:variable name="Record08">Weight {</txp:variable>

<txp:smd_each type="fixed" paramdelim="|" include="MeasureTypeCode|{MeasureTypeCode}" subset="2">

   <txp:smd_each type="fixed" paramdelim="|" include="Measurement|{Measurement}" subset="2" var_prefix="meas_">

      <!-- Only interested in the {smd_var_counter}th entry of the Measurement -->
      <txp:smd_if field="{smd_var_counter}" value="{meas_var_counter}">

         <!-- Concatenate the Measurement value. Record{smd_var_value} is the txp:variable name of the current MeasureTypeCode -->
         <txp:variable name='Record{smd_var_value}'><txp:variable name='Record{smd_var_value}' /> {meas_var_value}</txp:variable>

      </txp:smd_if>

   </txp:smd_each>

   <txp:smd_each type="fixed" paramdelim="|" include="MeasureUnitCode|{MeasureUnitCode}" subset="2" var_prefix="unit_">

      <!-- Only interested in the {smd_var_counter}th entry of the MeasureUnitCode -->
      <txp:smd_if field="{smd_var_counter}" value="{unit_var_counter}">

         <!-- Concatenate the MeasurenitCode value. Record{smd_var_value} is still the txp:variable name of the current MeasureTypeCode -->
         <txp:variable name='Record{smd_var_value}'><txp:variable name='Record{smd_var_value}' /> {unit_var_value} }</txp:variable>

      </txp:smd_if>

   </txp:smd_each>

</txp:smd_each>

<!-- Variables are now all built up; just display them for now -->
<ul>
<li><txp:variable name="Record01" /></li>
<li><txp:variable name="Record02" /></li>
<li><txp:variable name="Record03" /></li>
<li><txp:variable name="Record08" /></li>
</ul>

</txp:smd_xml>

Last edited by Bloke (2010-03-03 23:20:44)

mapu · 2010-03-12 20:41:54

How can make one a list of his Top Last.fm albums when the XML response looks like this:

<topalbums user="RJ" type="overall">
  <album rank="1">
    <name>Images and Words</name>
    <playcount>174</playcount>
    <mbid>f20971f2-c8ad-4d26-91ab-730f6dedafb2</mbid>  
    <url>
      http://www.last.fm/music/Dream+Theater/Images+and+Words
    </url>
    <artist>
      <name>Dream Theater</name>
      <mbid>28503ab7-8bf2-4666-a7bd-2644bfc7cb1d</mbid>
      <url>http://www.last.fm/music/Dream+Theater</url>
    </artist>
    <image size="small">...</image>
    <image size="medium">...</image>
    <image size="large">...</image>
  </album>
</topalbums>

Everything works fine until I’m trying to parse the nested fields and the <image size="..."> fields. Could someone give me a hint how to acomplish this, please? please?

Otherwise, another great plugin, Stef! Makes me feel again like a programmer! ;-)

Bloke · 2010-03-12 21:08:58

mapu wrote:

Everything works fine until I’m trying to parse the nested fields and the <image size="..."> fields.

I think you may need to wait until I’ve finished the nested rules portion of the code, sorry. I have a tentative version about 60% done, just been sidetracked the last few days.

Since lastfm are reusing things like name, mbid, and url you’ll need a version of the plugin that allows you to specify that you want fields="name, artist->name, artist->mbid" and so on. The plugin should be able to keep those separate so you can grab them separately from the feed. I’m also thinking about a way of allowing a shorthand so you don’t have to specify each and every sub-tag if you happen to want them all. Not sure if I can figure that out, but I’ll try.

I also need to be smarter with concatenation of like-named nodes. At the moment it doesn’t take attributes into account, but it should. Bad plugin *spank spank* no gruel for you…

Otherwise, another great plugin, Stef! Makes me feel again like a programmer! ;-)

Thanks, uhhh, I think ;-)

Last edited by Bloke (2010-03-12 21:09:23)

mapu · 2010-03-13 20:03:33

Then I will wait patiently for the new version! ;-)

nardo · 2010-03-30 00:22:07

Using Flickr API to get info back (due to limit on photos via RSS) … and having some issues with feed below

<rsp stat="ok">
<photos page="1" pages="1" perpage="500" total="109">
<photo id="444" owner="444" secret="444" server="2804" farm="3" title="Photo title" ispublic="1" isfriend="0" isfamily="0" ownername="Photo Owner Name" dateadded="1269854014" />
<photo id="555" owner="555" secret="444" server="2804" farm="3" title="Photo title" ispublic="1" isfriend="0" isfamily="0" ownername="Photo Owner Name" dateadded="1269854014" />

… etc …

</photos>
</rsp>

If I set attribute record as “photos” – I see one result (and replacement tags do nothing – i.e. don’t replace)
If I set attribute record as “photo” – I see 109 result (and replacement tags do nothing – i.e. don’t replace)

Is this data format not compatible with smd_xml due to the self-closing tags?

nardo · 2010-03-30 00:48:11

UPDATE – by requesting “extras” from the Flickr API, I now get the following:

<photo id="666" owner="666" secret="666" server="2698" farm="3" title="summer" ispublic="1" isfriend="0" isfamily="0" ownername="Owner Name" dateadded="1269796453" license="0" dateupload="1269751696" datetaken="2010-03-21 12:24:50" datetakengranularity="0" iconserver="2761" iconfarm="3" lastupdate="1269774620" latitude="0" longitude="0" accuracy="0" tags="summer" machine_tags="" views="1">
<description>summer</description>
</photo>

setting attribute record to “photo”, I can extract {description} … but not the other metadata within the “photo” tag …

Bloke · 2010-03-30 08:31:41

nardo wrote:

having some issues with feed

Hmmm, I tried setting the record to rsp as well and it still did nothing. It seems this isn’t a classical ‘record’ structure insofar as it’s just a list of self-closing tags with a tonne of attributes.

You’re right that smd_xml is a bit dim when it comes to self-closing tags. For reasons I have yet to uncover, the parser skips over them. I need to see if I can factor such tags into the new version.

You’ll get the same issues with the ‘extra’ data because the photo tag still doesn’t contain any text; just sub-tags. Again, I need to be smarter about how I capture such tags inside the plugin. Sorry.

jens31 · 2010-04-19 13:24:16

hii

thanks for this great(!!!!) plugin. makes live easy!!

but nevertheless, i have my issues ;)
is it possible to sort the output? i guess its now sorted by date, but it would be kool, to get it sorting by a ‘field’..?
possible?

Bloke · 2010-04-19 14:08:56

jens31 wrote:

is it possible to sort the output?

Nope, sorry. I’m not sure if the plugin can perform a sort because all the data is parsed and reformatted on-the-fly by your plugin form/container. I can’t predict how you’re going to use these; I’d have to be some kind of demi-God to sort free-form information ;-)

i guess its now sorted by date

It’s sorted by the order of the incoming XML data (I presume it’s date in your case).

One thing you could try is to output the data in tabular format and offer your visitors the option to sort it themselves using one of the many jQuery table sorting plugins out there. Never used them myself, but it might be worth a shot.

Textpattern CMS

Textpattern CMS support forum

#37 2010-02-22 09:30:34

Re: smd_xml : extract data from XML feeds

#38 2010-02-22 23:01:29

Re: smd_xml : extract data from XML feeds

#39 2010-03-01 20:38:29

Re: smd_xml : extract data from XML feeds

#40 2010-03-03 23:20:20

Re: smd_xml : extract data from XML feeds

#41 2010-03-12 20:41:54

Re: smd_xml : extract data from XML feeds

#42 2010-03-12 21:08:58

Re: smd_xml : extract data from XML feeds

#43 2010-03-13 20:03:33

Re: smd_xml : extract data from XML feeds

#44 2010-03-30 00:22:07

Re: smd_xml : extract data from XML feeds

#45 2010-03-30 00:48:11

Re: smd_xml : extract data from XML feeds

#46 2010-03-30 08:31:41

Re: smd_xml : extract data from XML feeds

#47 2010-04-19 13:24:16

Re: smd_xml : extract data from XML feeds

#48 2010-04-19 14:08:56

Re: smd_xml : extract data from XML feeds

Board footer