smd_remote_file: Manage remote URL downloads via TXP's Files tab

iblastoff · 2008-06-10 01:26:37

oh btw, it seems <txp:smd_file_download_link> works as a container tag but as a single tag (<txp:smd_file_download_link id=”#” />) it returns the url with the .link still at the end plus an an additional double quote at the end so it looks like

http://blahblah.com/file_download/id/band-song.mp3.link"

just a heads up!

Last edited by iblastoff (2008-06-10 01:27:26)

Bloke · 2008-06-10 08:36:14

iblastoff wrote:

<txp:smd_file_download_link> works as a container tag but as a single tag… it returns the url with the .link still at the end plus an an additional double quote at the end

Nuts, you’re right. Never tested it as a single tag and made a stupid assumption. Fixed in v0.43 | compressed

Thanks for spotting that.

hurty · 2008-06-11 18:07:24

I have a site with a couple thousand mp3 files that I need to move to be served from Akamai. Currently I’m just serving those files from the files directory on the local server. I’d like to update the file listings in the textpattern database (in the files table) using an SQL query so I don’t have to pfutz with the Texpattern files tab interface and make the change one file at a time.

Once I have the plugin installed (which I will do when I upgrade to TXP 4.0.6) can I just change the value in the filename field in the files table to my Akamai URL with .link appended to the filename? I.E. is it sufficient to change filename from:

“DCFC_DaytrotterSession_1.mp3”

to

“http://my.akamai-url.com/path/DCFC_DaytrotterSession_1.mp3.link”

??

Thanks! (changing a couple of thousand files, one at a time, using the TXP files tab is a task I could not imagine undertaking.)

Mark

Bloke · 2008-06-11 18:44:09

hurty wrote:

can I just change the value in the filename field in the files table to my Akamai URL with .link appended to the filename?

Sadly, no :-(

Since URLs are not exactly valid file names (and base64ing them made them too ugly), all remote files are stored as standard placeholder text files in the files directory. They take on the filename of your remote file + “.link”, i.e. “DCFC_DaytrotterSession_1.mp3.link”. Inside each file is the full URL “http://my.akamai-url.com/path/DCFC_DaytrotterSession_1.mp3”. Optionally you can specify alternate URLs — one per line — inside the text file so the load can be spread if required. I guess in this case, one place to manage a few thousand mp3s is enough!

But all is not lost. You could do a <txp:file_download_list category="somecat" /> and add some PHP trickery inside a files form. With each file, grab its file_download_name, create a file in the files dir with the same name + .link, write the URL to it (I’m assuming each TXP file category will be a separate dir on the remote server, hence the category filter in the file_download_list), close the file then update that particular txp_file entry in the database, appending .link to the filename already there. And go and make a cuppa while it’s working.

[ I suppose you could even arrange to have the script FXP the files across too if you were so bold ]

I think that’s all you’d need to do(!) Best thing is to create a dummy file using the smd_remote_file interface, check what it creates in txp_file and in the files dir, compare it to a regular file and then emulate the changes programmatically. I did plan to offer an extra tag to move a file to a URL and back again but never managed it; it was quite complex to make a generic process, sorry.

Needless to say, backup first and test it on a small subsection of files :-)

hurty · 2008-06-11 19:57:20

thanks for the quick response.

Just to clarify, then, the filename in files table in the textpattern database should be something like:

DCFC_DaytrotterSession_1.mp3.link

The file stored in the local files directory (“DCFC_DaytrotterSession_1.mp3.link”) is just a text file which contains the URL for the actual file, which in my case would be something like:

http://my.akamai.com/directory_path/DCFC_DaytrotterSession_1.mp3

Sounds pretty simple. I need to create an SQL query to change all the filenames listed in the files table, and a script to create a bunch of text files that have the proper URL in them.

Thanks Again!

Mark

Last edited by hurty (2008-06-11 19:58:23)

Bloke · 2008-06-11 20:04:05

Yup. Enjoy!

And if you fancy sharing your SQL/script here it might help others in the same boat. Thanks in advance.

hurty · 2008-06-11 21:54:58

I’m in the process of switching hosts, which is what precipitated the need for a third party to serve our MP3 files, so I haven’t actually implemented all the changes, but I have run a test on a small staging system to prove that everything will work. I added some conditions to my script and query because I only want to change the handling for MP3 files.

Here’s what I did:

First, to create the text files, that hold the URL for the actual link, I ran this PHP script (from a file in a directory I created in which to stage all the text files):

======

<?
$dir = "../files";
$filetype = 'mp3';
$dh = opendir($dir);
while (($file = readdir($dh)) !== false) {
 $type = substr($file,-3);
 if ($type == $filetype) {
       $doc = "$file.link"; 
       $handle = fopen($doc, 'w');
       $data = "http://our.newmediadomain.com/path/".$file; 
       fwrite($handle, $data); 
       print "Data Written to " . $doc ."<br />"; 
    }        
      fclose($handle); 
}
closedir($dh);
?>

====

Second I ran this SQL query on the files table:

====

 update file set filename = concat(filename,'.link') where (SUBSTRING(filename, -3) = 'mp3')

====

On my test batch, everything seemed to work just fine. If I run into problems on the production setup, I’ll post revisions.

Mark

Daytrotter

Bloke · 2008-06-11 21:56:35

Nice, thanks.

hurty · 2008-06-13 19:56:19

Still working with the staging server, but I now have the forms working with the plugin. Everything works like a charm, except that the counter does not increment as I download files. All the files are served from a single remote service, the status of each file is “Live.” Did I miss something that I need to configure? Each text file that contains the URL for the remote file is named:

SomeName_DaytrotterSession_N.mp3.link

Every text file contains the link to the actual MP3 file. The files download, and even play in my little flash previewer. But nothing increments the counter.

++++++

UPDATE: One other problem has reared it’s head. When I serve a file from the local server, the file is downloaded automatically to the user’s computer (which is the behaviour I expect from the file download mechanism within textpattern), but when I click the download link for the MP3 file that is being hosted on the remote server, it displays the MP3 in the browser window, and the user needs to manually save the file to their desktop.

Also, the counter increments as expected for local files — only the remote file downloads do not increment the counter.

++++++
2nd UPDATE: Another little quirk — the remote file size is not being accurately displayed, which could explain why the counter is not incrementing. (I noticed a validation routine in the plugin code which compares the reported size of the remote file to the downloaded file.)
Mark

Last edited by hurty (2008-06-13 22:20:09)

Bloke · 2008-06-14 08:38:07

hurty wrote:

the counter does not increment… Did I miss something that I need to configure?

Not as far as I can tell from your description; count_downloads is on by default so it… should… just… work. Let me check it still works on my test server and report back. I shouldn’t think my latest edit would affect it, and it was working before. Hmmm.

When I serve a file from the local server, the file is downloaded automatically, but when I click the download link for the MP3 file that is being hosted on the remote server, it displays the MP3 in the browser window, and the user needs to manually save the file to their desktop.

Interesting. I’d guess this behaviour is down to server/browser configuration. If somewhere along the line an .mp3 MIME type/extension has been associated with a media player plugin you’ll get this behaviour. If you wanna test this theory, rename one of the files on your server (and within the TXP interface) to have a bogus extension like .mp9 and then click the download link. Does it download it from both locations then?

Also, the counter increments as expected for local files — only the remote file downloads do not increment the counter.

Now that is totally bizarre because they call the same function (I think, from memory). Would you mind posting the links you see in the status bar when you hover over the download link from both a file on the admin screen and the same remote file from the public side? You can remove the domain.com to protect the guilty, it’s the bit afterwards I’m interested in :-)

Another little quirk — the remote file size is not being accurately displayed, which could explain why the counter is not incrementing.

Ah, right. Yes the file size in the TXP interface must match the true size of the file or it will not trigger the download. I figured since your files were already in TXP, the file size column would not need altering when you moved the files to a remote location.

There is a convoluted — but imo valid — reason for this behaviour, which I wonder if I should make an option in the smd_file_download tag (if I can hack it to do so; might require some brain power). Sometimes when you serve a file, if the remote host is down or just being awkward for whatever reason, some servers return a simple 403/404 which triggers TXP to pass that error on. Other servers return a fixed “file not found” image or some other obscure method of notification instead of the correct HTTP status. In those circumstances the act of sending the ‘doh’ image triggered the download counter to increment because it thought it was a valid HTTP 200 request.

Since the probability is rather low that the file size of a small image is the same as an mp3, comparing the size of the returned object before deciding to deliver the file seemed like a good idea at the time. Perhaps not.

The other thing I have noticed — and this could be considered a bug — is that if you update the info of a remote file in the TXP admin interface, it will look at the very first URL in the .link file and request its file size, which it then puts in the database. I had to do this because the core TXP behaves this way. Without me overriding the file size, TXP incorrectly (well, correctly, but we don’t want this) updates the file size to that of the .link itself, which is only a handful of bytes.

BUT, if the remote server is down or choked or being otherwise annoying, the file size request will fail. In that case you may receive a fugly error at the bottom of the files window and the file size will be set to 1 byte.

I haven’t figured out how to cleanly detect this situation yet and also what to do about it even if I can detect it with some timeout. I experimented but the problem is that, if it fails, TXP still insists on updating the file size to that of the .link file. I suppose I would have to grab the size from the database first just in case the remote update failed and if it did, replace the file size with the one that was there before. This doubles the number of queries — which I was trying to avoid — but since it’s only on file save I suppose we could deem this a necessary sacrifice. Edited to add: thinking about this, I probably do this anyway since I need to check the file sizes match. Been a long time since I looked at that bit of the code. Need a refresher.

If you can think of any other genius ideas on how to get round this, please feel free to share them. I’ll give it some thought as well and see if I can figure out why your counter’s not working.

Last edited by Bloke (2008-06-14 08:44:19)

hurty · 2008-06-14 14:16:28

Okay — I looked a little closer at the database, and with your helpful explanation, I can see why my counter isn’t working.

I had to upgrade to TXP 4.0.6 from 4.0.4 in order to be able to use the plugin. There are several new columns in the file table that did not exist in 4.0.4 — size, created, modified, and status. When I upgraded, TXP did not update the size column, so all the files have a size of “NULL.” If I manually update the .link file by overwriting the URL using the form field your plugin provides in the TXP admin interface, the size field is properly updated and downloads are then counted for that file.

What I need to do is find a way to force Textpattern to re-update the size column for all the rows in the table without having to go through the Admin UI and update each file, one-by-one. Any clues on how I might do that?

As for the download versus delivering the file to a browser window problem — I am convinced that it’s an issue with how the Akamai server is configured. I’m working on that problem from a different angle. If I find that there’s a way to finesse the issue from within Textpattern, I’ll report back.

Thanks, Bloke, for the help!

Mark

Bloke · 2008-06-14 14:38:50

Glad you (sort of) found the problem.

hurty wrote:

What I need to do is find a way to force Textpattern to re-update the size column for all the rows in the table without having to go through the Admin UI and update each file, one-by-one. Any clues on how I might do that?

Erm, ummm, errrr… ruud? :-) Anyone?

Sorry, no idea at the moment. If I have a brainwave I’ll yell.

Last edited by Bloke (2008-06-14 14:39:11)

hurty · 2008-06-14 17:09:04

okay — I couldn’t figure out a way to force textpattern to automatically update the size column in the file table, but it was easy enough to write a php script that grabbed the filename and the filesize from the filesystem and save the results of that script as an SQL query, which can insert the filesize for each file in the size column in the table.

Here’s that query:

  UPDATE txp_file SET size = CASE filename
      WHEN 'DCFC_DaytrotterSession_1.mp3.link' THEN '2129403'
      WHEN 'DCFC_DaytrotterSession_2.mp3.link' THEN '2759231'
          [ ... repeat 1500 times ...]
      ELSE size END

Which gets me most of the way to the goal. Last thing is to figure out how to force AKAMAI to honor the content disposition headers so that the file is delivered as a download and not displayed in a browser window.

Mark

Bloke · 2008-06-14 17:14:18

Nice!

Thanks for keeping us updated on your progress. The server config part is where I bow out graciously and leave it to people who understand stuff like that :-)

marazmus · 2008-08-20 04:01:22

Hello.

Can you add the “show and edit current (existing) remote link(s)” feature, please?

I mean, when I click “edit file”, Textpattern should show me current remote URL assigned to this file. And I can edit them.

Thank you.

Textpattern CMS

Textpattern CMS support forum

#31 2008-06-10 01:26:37

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#32 2008-06-10 08:36:14

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#33 2008-06-11 18:07:24

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#34 2008-06-11 18:44:09

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#35 2008-06-11 19:57:20

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#36 2008-06-11 20:04:05

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#37 2008-06-11 21:54:58

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#38 2008-06-11 21:56:35

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#39 2008-06-13 19:56:19

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#40 2008-06-14 08:38:07

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#41 2008-06-14 14:16:28

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#42 2008-06-14 14:38:50

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#43 2008-06-14 17:09:04

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#44 2008-06-14 17:14:18

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

#45 2008-08-20 04:01:22

Re: smd_remote_file: Manage remote URL downloads via TXP's Files tab

Board footer