smd_remote_file: Manage remote URL downloads via TXP's Files tab

hurty · 2008-06-11 21:54:58

I’m in the process of switching hosts, which is what precipitated the need for a third party to serve our MP3 files, so I haven’t actually implemented all the changes, but I have run a test on a small staging system to prove that everything will work. I added some conditions to my script and query because I only want to change the handling for MP3 files.

Here’s what I did:

First, to create the text files, that hold the URL for the actual link, I ran this PHP script (from a file in a directory I created in which to stage all the text files):

======

<?
$dir = "../files";
$filetype = 'mp3';
$dh = opendir($dir);
while (($file = readdir($dh)) !== false) {
 $type = substr($file,-3);
 if ($type == $filetype) {
       $doc = "$file.link"; 
       $handle = fopen($doc, 'w');
       $data = "http://our.newmediadomain.com/path/".$file; 
       fwrite($handle, $data); 
       print "Data Written to " . $doc ."<br />"; 
    }        
      fclose($handle); 
}
closedir($dh);
?>

====

Second I ran this SQL query on the files table:

====

 update file set filename = concat(filename,'.link') where (SUBSTRING(filename, -3) = 'mp3')

====

On my test batch, everything seemed to work just fine. If I run into problems on the production setup, I’ll post revisions.

Mark

Daytrotter

Bloke · 2008-06-11 21:56:35

Nice, thanks.

hurty · 2008-06-13 19:56:19

Still working with the staging server, but I now have the forms working with the plugin. Everything works like a charm, except that the counter does not increment as I download files. All the files are served from a single remote service, the status of each file is “Live.” Did I miss something that I need to configure? Each text file that contains the URL for the remote file is named:

SomeName_DaytrotterSession_N.mp3.link

Every text file contains the link to the actual MP3 file. The files download, and even play in my little flash previewer. But nothing increments the counter.

++++++

UPDATE: One other problem has reared it’s head. When I serve a file from the local server, the file is downloaded automatically to the user’s computer (which is the behaviour I expect from the file download mechanism within textpattern), but when I click the download link for the MP3 file that is being hosted on the remote server, it displays the MP3 in the browser window, and the user needs to manually save the file to their desktop.

Also, the counter increments as expected for local files — only the remote file downloads do not increment the counter.

++++++
2nd UPDATE: Another little quirk — the remote file size is not being accurately displayed, which could explain why the counter is not incrementing. (I noticed a validation routine in the plugin code which compares the reported size of the remote file to the downloaded file.)
Mark

Last edited by hurty (2008-06-13 22:20:09)

Bloke · 2008-06-14 08:38:07

hurty wrote:

the counter does not increment… Did I miss something that I need to configure?

Not as far as I can tell from your description; count_downloads is on by default so it… should… just… work. Let me check it still works on my test server and report back. I shouldn’t think my latest edit would affect it, and it was working before. Hmmm.

When I serve a file from the local server, the file is downloaded automatically, but when I click the download link for the MP3 file that is being hosted on the remote server, it displays the MP3 in the browser window, and the user needs to manually save the file to their desktop.

Interesting. I’d guess this behaviour is down to server/browser configuration. If somewhere along the line an .mp3 MIME type/extension has been associated with a media player plugin you’ll get this behaviour. If you wanna test this theory, rename one of the files on your server (and within the TXP interface) to have a bogus extension like .mp9 and then click the download link. Does it download it from both locations then?

Also, the counter increments as expected for local files — only the remote file downloads do not increment the counter.

Now that is totally bizarre because they call the same function (I think, from memory). Would you mind posting the links you see in the status bar when you hover over the download link from both a file on the admin screen and the same remote file from the public side? You can remove the domain.com to protect the guilty, it’s the bit afterwards I’m interested in :-)

Another little quirk — the remote file size is not being accurately displayed, which could explain why the counter is not incrementing.

Ah, right. Yes the file size in the TXP interface must match the true size of the file or it will not trigger the download. I figured since your files were already in TXP, the file size column would not need altering when you moved the files to a remote location.

There is a convoluted — but imo valid — reason for this behaviour, which I wonder if I should make an option in the smd_file_download tag (if I can hack it to do so; might require some brain power). Sometimes when you serve a file, if the remote host is down or just being awkward for whatever reason, some servers return a simple 403/404 which triggers TXP to pass that error on. Other servers return a fixed “file not found” image or some other obscure method of notification instead of the correct HTTP status. In those circumstances the act of sending the ‘doh’ image triggered the download counter to increment because it thought it was a valid HTTP 200 request.

Since the probability is rather low that the file size of a small image is the same as an mp3, comparing the size of the returned object before deciding to deliver the file seemed like a good idea at the time. Perhaps not.

The other thing I have noticed — and this could be considered a bug — is that if you update the info of a remote file in the TXP admin interface, it will look at the very first URL in the .link file and request its file size, which it then puts in the database. I had to do this because the core TXP behaves this way. Without me overriding the file size, TXP incorrectly (well, correctly, but we don’t want this) updates the file size to that of the .link itself, which is only a handful of bytes.

BUT, if the remote server is down or choked or being otherwise annoying, the file size request will fail. In that case you may receive a fugly error at the bottom of the files window and the file size will be set to 1 byte.

I haven’t figured out how to cleanly detect this situation yet and also what to do about it even if I can detect it with some timeout. I experimented but the problem is that, if it fails, TXP still insists on updating the file size to that of the .link file. I suppose I would have to grab the size from the database first just in case the remote update failed and if it did, replace the file size with the one that was there before. This doubles the number of queries — which I was trying to avoid — but since it’s only on file save I suppose we could deem this a necessary sacrifice. Edited to add: thinking about this, I probably do this anyway since I need to check the file sizes match. Been a long time since I looked at that bit of the code. Need a refresher.

If you can think of any other genius ideas on how to get round this, please feel free to share them. I’ll give it some thought as well and see if I can figure out why your counter’s not working.

Last edited by Bloke (2008-06-14 08:44:19)

hurty · 2008-06-14 14:16:28

Okay — I looked a little closer at the database, and with your helpful explanation, I can see why my counter isn’t working.

I had to upgrade to TXP 4.0.6 from 4.0.4 in order to be able to use the plugin. There are several new columns in the file table that did not exist in 4.0.4 — size, created, modified, and status. When I upgraded, TXP did not update the size column, so all the files have a size of “NULL.” If I manually update the .link file by overwriting the URL using the form field your plugin provides in the TXP admin interface, the size field is properly updated and downloads are then counted for that file.

What I need to do is find a way to force Textpattern to re-update the size column for all the rows in the table without having to go through the Admin UI and update each file, one-by-one. Any clues on how I might do that?

As for the download versus delivering the file to a browser window problem — I am convinced that it’s an issue with how the Akamai server is configured. I’m working on that problem from a different angle. If I find that there’s a way to finesse the issue from within Textpattern, I’ll report back.

Thanks, Bloke, for the help!

Mark

Bloke · 2008-06-14 14:38:50

Glad you (sort of) found the problem.

hurty wrote:

What I need to do is find a way to force Textpattern to re-update the size column for all the rows in the table without having to go through the Admin UI and update each file, one-by-one. Any clues on how I might do that?

Erm, ummm, errrr… ruud? :-) Anyone?

Sorry, no idea at the moment. If I have a brainwave I’ll yell.

Last edited by Bloke (2008-06-14 14:39:11)

hurty · 2008-06-14 17:09:04

okay — I couldn’t figure out a way to force textpattern to automatically update the size column in the file table, but it was easy enough to write a php script that grabbed the filename and the filesize from the filesystem and save the results of that script as an SQL query, which can insert the filesize for each file in the size column in the table.

Here’s that query:

  UPDATE txp_file SET size = CASE filename
      WHEN 'DCFC_DaytrotterSession_1.mp3.link' THEN '2129403'
      WHEN 'DCFC_DaytrotterSession_2.mp3.link' THEN '2759231'
          [ ... repeat 1500 times ...]
      ELSE size END

Which gets me most of the way to the goal. Last thing is to figure out how to force AKAMAI to honor the content disposition headers so that the file is delivered as a download and not displayed in a browser window.

Mark

Bloke · 2008-06-14 17:14:18

Nice!

Thanks for keeping us updated on your progress. The server config part is where I bow out graciously and leave it to people who understand stuff like that :-)

marazmus · 2008-08-20 04:01:22

Hello.

Can you add the “show and edit current (existing) remote link(s)” feature, please?

I mean, when I click “edit file”, Textpattern should show me current remote URL assigned to this file. And I can edit them.

Thank you.

Bloke · 2008-08-20 11:12:27

marazmus wrote:

Can you add the “show and edit current (existing) remote link(s)” feature, please?

It’s a nice idea and I’d like to offer that, but I don’t know how to do it. My early attempts when initially writing the plugin failed :-( Since it’s not possible to edit normal ‘local’ files from the TXP interface, do you know any way I can make remote files behave differently? At no place in the current interface is there anywhere that allows you to download the file from the files directory, edit it and re-upload it automatically, nor a facility to edit the file in situ.

Off the top of my head, the only option open to me is to (somehow) read the contents of the .link file and display each URL in a list (a column of text boxes?), somewhere in the edit window. Then, if you edit the URLs, they are written back to the file again when you hit ‘save’. Normally, any edits in that window only change what’s stored in the database so to edit the files as well is not a simple change.

The other problem with that approach is that the remote filename is tied to the local file; the filenames MUST match or it all falls apart (it’s the same reason you cannot edit the filename of the file once it’s uploaded; the file itself is not touched, only the database details that relate to the file are updated). If I could restrict editing to the ‘path’ part of the URL only, that would perhaps be better. Might be a way of doing that universally, if I put my mind to it but I’m not sure at this stage.

At the moment, the simplest thing to do is probably to use a text editor that is capable of editing and auto-re-uploading files on servers. Then you can open the .link files directly from your desktop, edit the URLs and hit save, triggering the upload.

Of course, if you or anyone else can think of a way of handling this situation cleanly in the plugin, then please let me know. It would be a very useful feature.

marazmus · 2008-08-20 11:39:45

Bloke wrote:

Off the top of my head, the only option open to me is to (somehow) read the contents of the .link file and display each URL in a list (a column of text boxes?), somewhere in the edit window. Then, if you edit the URLs, they are written back to the file again when you hit ‘save’. Normally, any edits in that window only change what’s stored in the database so to edit the files as well is not a simple change.

You’re absolutely right. Ok, I just will try to explain my situation. I have many large files, and uploading they via ftp only. No “inner” uploads from Textpattern, because they are large (up to 500 Mb). And the tasks steps is simple:

1) upload file via ftp
2) create new file link in Textpattern and use remote URL via your plugin
3) publish link in article or news or elswhere in this Textpattern site

Okay, let’s differ situation. Now we have another server (mirror) for our files. We should use remote URL’s in ONE file link. But I can’t see from Textpattern admin side what link’s IS used. Only go to ftp and see .link file manually. We need there just a way to parse .link file, create inputs for remote URL’s and fill them with URL’s that .link file contains. Then edit and save they to .link file again.

The other problem with that approach is that the remote filename is tied to the local file; the filenames MUST match or it all falls apart (it’s the same reason you cannot edit the filename of the file once it’s uploaded; the file itself is not touched, only the database details that relate to the file are updated). If I could restrict editing to the ‘path’ part of the URL only, that would perhaps be better. Might be a way of doing that universally, if I put my mind to it but I’m not sure at this stage.

Yes, “edit path” feature. Yes, this is mean that you really know what you do when changing the path of the file, but this is not plugin problem :)

At the moment, the simplest thing to do is probably to use a text editor that is capable of editing and auto-re-uploading files on servers. Then you can open the .link files directly from your desktop, edit the URLs and hit save, triggering the upload.

Got it :) I’m using it, but changing paths from inside Textpattern will be really cool :)

Of course, if you or anyone else can think of a way of handling this situation cleanly in the plugin, then please let me know. It would be a very useful feature.

I’m not the programmer, only “txp tag user” :) But maybe some programmers will see our problem and help us.

p.s. Way to change path of file links will give us simple download system. Not so monsteriously like PAFileDB or OLate Download, but functionally cool. Upload file from Txp inside, or upload file via ftp and use remote URL. If remote URL was broken (site is down, at example), just place file to another server and change path.

Bloke · 2008-08-20 11:56:13

marazmus

Thanks for clarifying. I think we are on the same page. Yes, it would be cool to do this. Yes I’d like to do it now I’ve thought about it these past few minutes. Yes, if I can figure it out, I will do it. When that takes place I currently cannot say, sorry. I have this new plugin hanging round my neck which desperately needs finishing, some updates to smd_if planned and I really want to revisit smd_query and smd_vars to get them roadworthy. After that I’ll probably look at this.

Yes, “edit path” feature. Yes, this is mean that you really know what you do when changing the path of the file, but this is not plugin problem :)

True, I could put it as a disclaimer, but since TXP itself disallows editing of the filename I should probably do likewise. Will think about it.

If remote URL was broken (site is down, at example), just place file to another server and change path.

Yes. But with a lot of files, this is cumbersome. The ability to add more than one URL and pick randomly was a nice idea I had but it’s incomplete. I did start work a long time ago on a companion plugin called smd_ping that checked remote files for a (set) timeout period and if no response was received would take one action or another. I intended to either use that or build something similar into this plugin so the choice of server was not entirely random but based on a fixed, random, start point and a test for “is the server up”. If not it’d try another random server until it found one that worked or exhausted the list. Yet again, it never made it past dev status before I moved on. One day…!

Textpattern CMS

Textpattern CMS support forum

#37 2008-06-11 21:54:58

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#38 2008-06-11 21:56:35

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#39 2008-06-13 19:56:19

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#40 2008-06-14 08:38:07

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#41 2008-06-14 14:16:28

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#42 2008-06-14 14:38:50

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#43 2008-06-14 17:09:04

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#44 2008-06-14 17:14:18

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#45 2008-08-20 04:01:22

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#46 2008-08-20 11:12:27

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#47 2008-08-20 11:39:45

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#48 2008-08-20 11:56:13

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

Board footer