[plugin] jk_parse_xml: get data from XML feeds

variaas · 2008-01-02 05:09:40

I just found this plugin and it looks great for some of my needs. If I understand correctly, you don’t cache the file yourself when pulling them, but recommend using a cron job to cache the file. I was wondering if you were planning to roll in some caching mechanism into the plugin itself. That way it’ll attempt to pull a local copy first, if not available or older than x hours it will go out and grab the latest from the external site.

Gocom · 2008-01-02 05:23:34

Caching is made to be easy. It’s all about choises. For helping the overloading society ;)

Cheers!

Last edited by Gocom (2008-01-02 05:25:08)

variaas · 2008-01-02 18:23:48

Thanks Gocom, this looks interesting, but I’m not sure how to implement. I use shared hosting and if I’m reading correctly I don’t have access to a memcache server.

I’m wondering if there is anything wrong with the cache solution I mentioned. So the first request for the xml will just pull the xml file (via CURL or get_file_content) and then write the file to a local file. Then you just reference that file every time until a set amount of time has passed and you just refresh the file.

Jeff_K · 2008-01-03 23:56:33

Hey, yeah. good suggestions! I will investigate when I get a second…

variaas · 2008-01-04 08:34:05

Jeff – I wrote up the cache feature myself for you to review – jk_parse_xml_v0.3a.php. It caches both the xml and images (jpg, bmp, gif, png) to your textpattern/cache/ folder. Let me know your thoughts.

sthmtc · 2008-01-04 08:50:13

variaas wrote:

Jeff – I wrote up the cache feature myself for you to review – jk_parse_xml_v0.3a.php. It caches both the xml and images (jpg, bmp, gif, png) to your textpattern/cache/ folder. Let me know your thoughts.

this is somewhat off-topic but wouldn’t it be nice to combine your asv_tumblelog plugin with the additional xml parsing power of jk_parse_xml? just a thought…

variaas · 2008-01-08 02:32:21

stmtc – that’s a good idea but out of curiosity does the new xml features of php5 make SimplePie somewhat obsolete?

Vitruvius · 2008-03-09 09:41:32

I’ve finally got around to testing this plugin (the project was put on hold until now)… but I am getting the same error as sthmtc

tag_error <txp:jk_parse_xml xml="http://www.eventbrite.com/rest/organizer_list_events/15857229" form="xml-test" local_file="0" parents="events" elements="title" limit="5" /> ->  Warning: DOMDocument::loadXML() [function.DOMDocument-loadXML]: Start tag expected, '<' not found in Entity, line: 1  on line 47

I’ve tried to set the useragent (using my some borrowed code) as sthmtc which is perhaps the problem? I’ve updated the cURL part of your plugin to read:

  // else if remote, and remote fopen off, try curl
  elseif (ini_get('allow_url_fopen') == 0) {
    $ch = curl_init();
    $timeout = 5; // set to zero for no timeout
    // set user agent
    // spoofing random agent
    $useragent="Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/521.25 (KHTML, like Gecko) Safari/521.24";
    curl_setopt($ch, CURLOPT_USERAGENT, $useragent);  
    curl_setopt ($ch, CURLOPT_URL, $xml);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
    $xml = curl_exec($ch);
    curl_close($ch);
  }

I’d really appreciate your assistance.

I’d be happy to financially contribute to you or your fantastic site if you are still around to work on this and maybe get 0.3a out as a 0.4 release??

SH

Last edited by Vitruvius (2008-03-09 09:42:23)

Jeff_K · 2008-03-10 06:04:29

Hi Vitruvius.

I can’t fully test out the caching plugin because Curl is throwing a server error on Dreamhost right now (?), but I tried loading (a trimmed down version of) your XML from a local file.. I got it to work, although I needed to switch “events” to “event”, based on looking at the file xml you are linking too.

I will see if I can get Curl up and running again on my end, and try adding the caching feature as mentioned above (Thanks!) and see if I can replicate your problem.. In the meantime, can you try loading the plugin fresh, and change parent from “events” to “event” and see if that helps?

Also, try loading the file locally and see if that works.

The error looks like an XML error…. That is a huge huge xml file to parse with DOMDocument::loadXML. I am not sure, but that might be causing some trouble. Its 156k and 2000+ lines from the looks of it.

Thanks!

Vitruvius · 2008-03-10 09:20:20

Hi Jeff,

I’ve had a good experiment with various things and have had some success and some weird results.

re: the size of the XML file – yes – its a worry – but ATM thats all that is on offer. That’s why I’d be dead keen to get a cached version happening. I may have to resort to a cron job to update a local file every now and again.

Here are the results of my testing:

Current plugin

The plugin is working with a local file (obviously with local_file = 1)
Yes – you were right – I changed “events” to “event” and that worked
The plugin works with a remote file BUT only when I have the variable local_file still set to 1 ??? I have double and triple checked this.
I can tell its loading the remote file due to the delay and the fact that I deleted any copy of the local file during testing
I still need to fake a user agent code – so I presume it is using cURL

Cached plugin

I tried the cached plugin (0.3a) and get a ton of errors complaining of a missing file.
I tried inserting the fake user agent – still no luck…

I’m really interested to see if you can make sense of all of that :)

SH

Jeff_K · 2008-03-10 16:41:38

I can’t get it to replicate the your problem with the local_file variable.. Can you try version .31 and let me know if that still causes problems.

I am just sorting through the caching now…

Jeff_K · 2008-03-10 19:21:15

Scratch that. I released version .4 which included file caching. Many thanks to everyone here for input and code!

Vitruvius, can you give it a shot and see if it works for you? Seems pretty speedy for me even with the 156k file.

Vitruvius · 2008-03-10 21:58:49

Hi Jeff,

Thank you very much for your work on this!

I tried version 0.4 and get the following error when trying to load an external file:

tag_error <txp:jk_parse_xml xml="http://www.eventbrite.com/rest/organizer_list_events/15857229" form="xml-test" local_file="0" parents="event" elements="title,start_date,end_date,city,link" limit="100" wraptag="div" break="p" /> ->  Warning: fopen() expects at least 2 parameters, 1 given  on line 44
tag_error <txp:jk_parse_xml xml="http://www.eventbrite.com/rest/organizer_list_events/15857229" form="xml-test" local_file="0" parents="event" elements="title,start_date,end_date,city,link" limit="100" wraptag="div" break="p" /> ->  Warning: implode() [function.implode]: Bad arguments.  on line 4

Tested a local file and had no problems.

Any ideas?

SH

Last edited by Vitruvius (2008-03-10 21:59:04)

Jeff_K · 2008-03-10 22:07:13

Try .41. Looks like you have allow_url_fopen turned on, which means it shouldn’t need to use curl..

Vitruvius · 2008-03-11 00:35:15

Hi Jeff,

I hope you can’t tell that I really don’t know too much about server stuff! fopen, cURL it all sounds good to me :)

Tried 0.41, got only one error (and no output)

tag_error <txp:jk_parse_xml xml="http://www.eventbrite.com/rest/organizer_list_events/15857229" form="xml-test" local_file="0" parents="event" elements="title,start_date,end_date,city,link" limit="100" wraptag="div" break="p" /> ->  Warning: implode() [function.implode]: Bad arguments.  on line 45

Thanks!

SH

Textpattern CMS

Textpattern CMS support forum

#16 2008-01-02 05:09:40

Re: [plugin] jk_parse_xml: get data from XML feeds

#17 2008-01-02 05:23:34

Re: [plugin] jk_parse_xml: get data from XML feeds

#18 2008-01-02 18:23:48

Re: [plugin] jk_parse_xml: get data from XML feeds

#19 2008-01-03 23:56:33

Re: [plugin] jk_parse_xml: get data from XML feeds

#20 2008-01-04 08:34:05

Re: [plugin] jk_parse_xml: get data from XML feeds

#21 2008-01-04 08:50:13

Re: [plugin] jk_parse_xml: get data from XML feeds

#22 2008-01-08 02:32:21

Re: [plugin] jk_parse_xml: get data from XML feeds

#23 2008-03-09 09:41:32

Re: [plugin] jk_parse_xml: get data from XML feeds

#24 2008-03-10 06:04:29

Re: [plugin] jk_parse_xml: get data from XML feeds

#25 2008-03-10 09:20:20

Re: [plugin] jk_parse_xml: get data from XML feeds

#26 2008-03-10 16:41:38

Re: [plugin] jk_parse_xml: get data from XML feeds

#27 2008-03-10 19:21:15

Re: [plugin] jk_parse_xml: get data from XML feeds

#28 2008-03-10 21:58:49

Re: [plugin] jk_parse_xml: get data from XML feeds

#29 2008-03-10 22:07:13

Re: [plugin] jk_parse_xml: get data from XML feeds

#30 2008-03-11 00:35:15

Re: [plugin] jk_parse_xml: get data from XML feeds

Board footer