[solved] file_download regression in 4.5.5

minusf · 2014-01-18 03:20:32

it seems there is a regression in 4.5.5 for file_download
some of our old download links stopped working (returning 404).

for example:
http://www.sk-spell.sk.cx/file_download/53/Slovak-Thesaurus.oxt

this worked fine before (it has 185k downloads) and the file exists:
http://www.sk-spell.sk.cx/files/Slovak-Thesaurus.oxt

i have narrowed down the problem to filenames that contain
hyphans AND capitals, e.g.:

http://www.sk-spell.sk.cx/file_download/87/LanguageTool-1.0.1-dev.oxt
http://www.sk-spell.sk.cx/files/LanguageTool-1.0.1-dev.oxt

http://www.sk-spell.sk.cx/file_download/73/OOo-Thesaurus2-sk_SK.zip
http://www.sk-spell.sk.cx/files/OOo-Thesaurus2-sk_SK.zip

and so on.

other files work as expected.
is anyone else seeing this?

Last edited by Gocom (2014-03-30 09:48:22)

wet · 2014-01-18 07:06:00

I cannot reproduce this behaviour on a fresh installation of Textpattern 4.5.5.

Can you please provide more details:

Does the “File directory path” in Advanced preferences point to the correct directory on the server?
Which version of Textpattern were you running when the files in question were originally uploaded?
Which version of Textpattern was the last one you used when downloads worked ok?
Is the filename as it is displayed in the Content > Files tab exactly identical (including punctuation and case) to the one you see in the download URL?

minusf · 2014-01-18 08:55:54

the file “directory path” is correct and all other lowercase
downloads work as expected.

the site started in 2005, and it has been txp from the start.
i really can’t remember what version we were at when the files
were uploaded, it was years ago. the download has always worked.

the last known version was 4.4ish

the “[Download]” link in the “show detail” view on the
Files page points exactly to the same location:
http://www.sk-spell.sk.cx/file_download/53/Slovak-Thesaurus.oxt

i have tried turning off every plugin, same result.

minusf · 2014-01-18 09:22:44

all these mixed case files show “Condition: OK” on the files page.
very strange problem.

minusf · 2014-01-18 10:21:50

i forgot to add a very important detail, ALL the files can be downloaded
if the file name is left off from the end.

does not work:
http://www.sk-spell.sk.cx/file_download/53/Slovak-Thesaurus.oxt
works:
http://www.sk-spell.sk.cx/file_download/53

does not work:
http://www.sk-spell.sk.cx/file_download/87/LanguageTool-1.0.1-dev.oxt
works:
http://www.sk-spell.sk.cx/file_download/87

does not work:
http://www.sk-spell.sk.cx/file_download/73/OOo-Thesaurus2-sk_SK.zip
works:
http://www.sk-spell.sk.cx/file_download/73

this rules out any file permission, incorrect path issues.
all lowercase files work in all cases.

.htaccess has nothing added to it.

minusf · 2014-01-18 10:47:18

i have put some logging in publish.php:

       // we are dealing with a download
        error_log("file_error='$file_error'");
        if (@$s == 'file_download') {
                callback_event('file_download');
                if (!isset($file_error)) {

and this produces in apache’s error_log the following:

[Sat Jan 18 11:44:53 2014] [error] [client 171.98.40.117] file_error='404'

it is 404 even before getting to check if the file exists.

minusf · 2014-01-18 11:20:05

it seems that this part is the culprit (publish.php):

309                                         case urldecode(strtolower(urlencode(gTxt('file_download')))):
310                                                 $out['s'] = 'file_download';
311                                                 $out['id'] = (!empty($u2)) ? $u2 : '';
312                                                 $out['filename'] = (!empty($u3)) ? $u3 : '';
313                                                 break;

here is the $out array after this case is hit:

Array
(
    [status] => 200
    [id] => 53
    [s] => file_download
    [c] => 
    [context] => 
    [q] => 
    [m] => 
    [pg] => 
    [p] => 
    [month] => 
    [author] => 
    [request_uri] => /file_download/53/Slovak-Thesaurus.oxt
    [qs] => 
    [subpath] => \/
    [req] => /file_download/53/Slovak-Thesaurus.oxt
    [filename] => slovak-thesaurus.oxt
)

notice how the filename is incorrect.
i think this broke it:

http://code.google.com/p/textpattern/source/detail?r=3722&path=/development/4.x/textpattern/publish.php

wet · 2014-01-18 11:46:45

It looks like the database on your server has an uncommon setup with regard to collations.

This might be related to the fact that your database was probably created back in the dark ages when Textpattern had to support international character sets on MySQL version 4, i.e. lacking support for UTF-8, and never converted to UTF-8 later on.

New installs default to utf8_general_ci, i.e. they ignore case for string comparisons.

To verify, please try this SQL command in the MySQL client of your choice:

SELECT DEFAULT_COLLATION_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='your_textpattern_database_name';

Substitute ‘your_textpattern_database_name’ with the actual value from $txpcfg['db'] in config.php.

plus

SHOW CREATE TABLE prefix_txp_file;

Substitute ‘prefix_’ with the actual value from $txpcfg['table_prefix'] in config.php.

What’s the value of $txpcfg['dbcharset'] in config.php?

minusf · 2014-01-18 12:02:33

DEFAULT_COLLATION_NAME = utf8_general_ci
$txpcfg['dbcharset'] = 'utf8';

       Table: tp_txp_file
Create Table: CREATE TABLE `tp_txp_file` (
  `id` int(11) NOT NULL auto_increment,
  `filename` varchar(255) collate utf8_bin NOT NULL default '',
  `title` varchar(255) collate utf8_bin default NULL,
  `category` varchar(255) collate utf8_bin NOT NULL default '',
  `permissions` varchar(32) collate utf8_bin NOT NULL default '0',
  `description` mediumtext collate utf8_bin NOT NULL,
  `downloads` int(4) unsigned NOT NULL default '0',
  `status` smallint(6) NOT NULL default '4',
  `modified` datetime NOT NULL default '0000-00-00 00:00:00',
  `created` datetime NOT NULL default '0000-00-00 00:00:00',
  `size` bigint(20) default NULL,
  `author` varchar(255) collate utf8_bin NOT NULL default '',
  PRIMARY KEY  (`id`),
  UNIQUE KEY `filename` (`filename`),
  KEY `author_idx` (`author`)
) ENGINE=MyISAM AUTO_INCREMENT=104 DEFAULT CHARSET=utf8 COLLATE=utf8_bin PACK_KEYS=0

wet · 2014-01-18 12:51:32

minusf wrote #278262:

Table: tp_txp_file...
`filename` varchar(255) collate utf8_bin NOT NULL default '',

utf8_bin is problematic. As the filename column is collated with an utf8_bin order, lower-case and upper-case letters in filenames are considered different by MySQL.

Textpattern converts the requested URL into lower-case and (since 4.5.x, as you rightly pointed out), checks whether the filenames are matching if they are present in the requested URL. In a stock setup Textpattern employs a utf8_general_ci collation and thus instructs MySQL to ignore case in string comparisons.

You might want to dig into the reasons why your site was setup this particular way and then consider your options, or simply try to change the collation of the offending column back to utf8_general_ci and see how it goes:

ALTER TABLE tp_txp_file CHANGE COLUMN filename filename VARCHAR(255) NOT NULL DEFAULT '' COLLATE 'utf8_general_ci'

Make backups.

minusf · 2014-01-18 12:57:40

i am not sure there is a “particular” reason for the divergence.
its a quite old site started with an ancient txp.

i will compare a stock txp install with our current db and try to sync it up.

thank you for looking into this. case closed.

Textpattern CMS

Textpattern CMS support forum

#1 2014-01-18 03:20:32

[solved] file_download regression in 4.5.5

#2 2014-01-18 07:06:00

Re: [solved] file_download regression in 4.5.5

#3 2014-01-18 08:55:54

Re: [solved] file_download regression in 4.5.5

#4 2014-01-18 09:22:44

Re: [solved] file_download regression in 4.5.5

#5 2014-01-18 10:21:50

Re: [solved] file_download regression in 4.5.5

#6 2014-01-18 10:47:18

Re: [solved] file_download regression in 4.5.5

#7 2014-01-18 11:20:05

Re: [solved] file_download regression in 4.5.5

#8 2014-01-18 11:46:45

Re: [solved] file_download regression in 4.5.5

#9 2014-01-18 12:02:33

Re: [solved] file_download regression in 4.5.5

#10 2014-01-18 12:51:32

Re: [solved] file_download regression in 4.5.5

minusf wrote #278262:

#11 2014-01-18 12:57:40

Re: [solved] file_download regression in 4.5.5

Board footer