FTP server hit 10000 files limit in images folder...

kirito · 2017-02-02 17:01:48

Hello everybody,
I’ve set up a blog for my wife, who loads lots (and I say “lots”) of pictures. I’ve moved the blog from an old one where there were more than 6.000 pictures to an altervista free hosting. Just finished and ready to run, already posted a couple of articles. Then I discovered this: altervista ftp server doesn’t show up more than 10.000 files in the same directory. With more than 6.000 pictures and thumbnail there already are more than 12.000 files.

I can upload files, but I can’t backup the images folder. Any FTP client stop exactly at 10.000 files while mirroring. I suppose I can use the txp_image informations to generate a list of images and download every single file with a script, but I’m not so happy about this. Maybe I also could store images somewhere else, but I’m not so sure how should I do, and also I can’t complicate my wife workflow, because, you see, she’s not so “skilled” with this stuff…

Then the question is: is there a way to split images folder so that no more than 10.000 files are stored in each folder? Or can someone think of a simple way to workaround this ftp limit?

Thank you for any help.

bici · 2017-02-02 20:14:43

I dont have a solution for you.

But I have thought for awhile now that it would be nice to have more than one place to load images to. With EE i can specify file folder for images relating to specific sections and i find this very convenient for managing images by sections on my website rather than have one large IMAGE folder of all images.

ruud · 2017-02-02 20:36:29

To work around the problem, you’d have to store an image like 91827.jpg in a folder structure like this: 9/91827.jpg or even 9/1/91827.jpg

This requires editing core TXP files:
textpattern/include/txp_image.php
textpattern/lib/txplib_misc.php
textpattern/lib/class.thumb.php

To translate to the correct structure, you need an extra function, which can be added at the end of textpattern/lib/txplib_misc.php

function img_level($id)
{
   $level = 2; // how many levels deep the folder structure should be.
   return join(DS, str_split(substr(str_pad($input, $level, "0", STR_PAD_LEFT), 0, $level))).DS.$id; 
}

In textpattern/lib/txplib_misc.php edit the function image_data and replace $newpath = IMPATH.$id.$ext; with $newpath = IMPATH.img_level($id).$ext;

In textpattern/lib/class.thumb.php replace all occurrences of IMPATH.$this->m_id with IMPATH.img_level($this->m_id)

In textpattern/include/txp_image.php replace all occurences of IMPATH.$id with IMPATH.img_level($id)

This takes care of all the file paths, but not of the retrieval of the images via HTTP and because you probably don’t want to edit all the articles containing references to these images, it’s easier to add some rewrite rules in the .htaccess file

RewriteRule "^/image/(\d)(\d)(.*)$" "/path/to/images/$1/$2/$1$2$3"
RewriteRule "^/image/(\d)(\..*)$" "/path/to/images/0/$1/$1$2"

And of course you’d have to move all the existing images to their new file locations. Create a file called move.php in your images directory containing the following code and visit it via your browser:

<?php
$files = scandir('.');
foreach($files as $file) {
  if (!is_file($file)) continue;
  $id = str_pad(intval($file), 2, "0", STR_PAD_LEFT);
  if (!is_dir($id[0])) mkdir($id[0]);
  if (!is_dir($id[0].'/'.$id[1]) mkdir($id[0].'/'.$id[1]);
  rename($file, $id[0].'/'.$id[1].'/'.$file);
}
echo "done";

Something like that. All of this is not tested at all btw, but have fun with it and let us know if it works.

colak · 2017-02-02 21:14:47

it would be nice if images were saved in folders named after their categories.

ruud · 2017-02-02 21:30:26

colak wrote #303807:

it would be nice if images were saved in folders named after their categories.

If you changed the category for an image, you’d have to update all the links to that image.

bici · 2017-02-02 22:48:12

colak wrote #303807:

it would be nice if images were saved in folders named after their categories.

i prefer the freedom to create folders and call the images from folders in addition to the default ImageFolder. IS this something that could be accommodated in TxP Core?

Bloke · 2017-02-02 23:57:59

Bizarrely, just yesterday, I started thinking about a solution to this. I wrote some code last year, but forgot about it so I’m rethinking it now in the light of helping out a few people recently with massive image pools.

My approach is more as enabler than concrete implementer. To that end, I propose a callback in the imagesrcurl() function that will expose all the metadata about the file. A plugin could then (for example) take the image uploaded date and store images in subfolders based on year/month of upload. Or use the ID, as Ruud indicated above, slicing off the first (or last) one or two digits and making subdirs in the images directory. Entirely up to the plugin, though something immutable (its ID or upload date, currently) is most viable without creating additional burden.

For retrieval we need some way to intercept and alter IMPATH. In core this might take the form of a new companion function to imagesrcurl() which would also expose a callback hook. The default output of that function would be IMPATH as it is today, but any plugin could intercept and reroute requests to the structure as defined by the plugin.

Those two functions/callbacks ought to cover the entire process, from upload to update to retrieval or deletion of all images.

The imagesrcurl() function and its friend also need to become cognisant of image ‘flavour’, for want of a better word. Currently the 3rd argument to the function is a boolean: true = thumbnail, false = main image. This needs extending because in 4.7.0 we’re planning on auto-creating a third type of image: grid.

Missing grid images are probably going to be created on-the-fly when you switch to grid view, and cached in a sub-folder off the /images directory. I wouldn’t want to mass-create them, nor would I want to burden some people’s already bursting /images directory by creating them directly in there. During image upload/replacement the grid image could be created at that point, as thumbnails are today.

To take additional pressure off, we might use this opportunity in 4.7.0 to move thumbnails (again, on demand when they are viewed) to a sub-folder too, instead of just appending ‘t’ to them in the main images directory. Not sure if that’s possible cleanly yet without having to check in two places for the time being.

I think between the callbacks and some judicial use of convention, we can make image handling more streamlined, both in terms of how the UI handles it and also how the images are ultimately stored.

If anyone has any thoughts on this, some better ideas, or even some code to help make this a reality, it’d be incredibly beneficial. Practical image management has been overlooked for far too long in Txp. It works fine for a few hundred or a thousand images, but anyone with more demanding needs ought to be able to go further without bringing the server or user experience to its knees.

Last edited by Bloke (2017-02-03 00:02:20)

kirito · 2017-02-03 08:05:52

Well, I’m impressed. :)

I like to have all images stored in one dir, which makes my life easy for massive import new/replaced images and my other bash scripts (don’t laugh, please :D ). And I’m also always sure that if an image exists, it has to be there and nowhere else. I agree with Stef (Bloke) that multiple folders should be enabled, but not default (if I correctly understand his post: my english is far from perfect).

Anyway, for the moment I managed to download all files with wget, extracting a list of all images from a csv formatted txp_image table in the form “1234.ext”, one per line, with awk:

awk -F "'" '{ if ($22 == 1) { print $2 $8"\n"$2"t" $8 } else { print $2, $8 } }' txp_image.csv > imagelist.txt

So I get a list of images and thumbnails (if any) like this:

1.jpg
1t.jpg
2.png
3.gif
3t.gif
...

Then download with this:

xargs -i wget --user=username --password=password 'ftp://ftp.example.com/blog/images/{}' -nH -N -c --cut-dirs=1 < imagelist.txt

This is not perfect, because it download only new images, while modified ones being keep. But works, even for 14.000+ files like in my case. I tried with --mirror, but it becomes very, very, very slow. Anyway, packing this in a small script and add to cron should do the trick for backup purpose.

I would like to try ruud solution (terrific! I’m impressed, really!). I dont’t like to play with core files, but I’ll try if I have time to.

I would like to suggest something, but I really don’t understant implications of all those “callbacks”, “hooks” and whatsoever. I think thumbnails in separate folders would be good: it roughly halve the number of files per directory and would make it easy to implement multiple size thumbnails/images. That would be good to serve different size image for different devices/purposes.

Bloke · 2017-02-03 08:58:18

kirito wrote #303817:

multiple folders should be enabled, but not default (if I correctly understand “his post…)

That is correct. It might be possible to move thumbnail images (currently ending in ‘t’) and grid images to their own subdirectories. But that’s a big ‘if’.

I managed to download all files with wget, extracting a list of all images from a csv formatted txp_image table in the form “1234.ext”, one per line, with awk:

I love awk! And this post proves why. It’s so versatile for scripting.

I think thumbnails in separate folders would be good: it roughly halve the number of files per directory and would make it easy to implement multiple size thumbnails/images. That would be good to serve different size image for different devices/purposes.

There’s already a plugin for that, btw ;-)

But it did occur to me that if we’re upgrading imagesrcurl() to become aware of different ‘types’ of image (thumbnail, grid) then we could display them both on the Image Edit panel. You can already upload/replace thumbnails and they’ll automatically be recreated from the main image if you replace that. Why not extend that to grid images too? And for that matter, why not add a ‘+ button to allow you to create a new ‘type’. Name it, give it dimensions, store them in the database (somewhere) and then when you go to upload/replace an image you’ll get one automatically made at the new dimensions too, which you can manually replace if you want later.

Perhaps viewing an image without one of these new ‘custom types’ will auto-create it, I don’t know. We’d need an upgrade to <txp:thumbnail> to allow you to specify a ‘type’ (default: type="thumbnail") but after that, everything should work fine and allow you to manage arbitrary thumbnails.

The callbacks/hooks I propose would then simply allow a plugin to step in and offer the ability to make further sub-divisions of each ‘type’ (including the main image directory) to limit the number of files in each of the ‘type’ sub-directories.

Not sure. But I’d like to explore this idea as multiple image sizes is becoming a more common occurrence in the days of retina and high-definition displays. If anyone can see a decent path to achieve that then please weigh in. Or if anyone can submit code / a PR for it, I’d be very grateful.

jakob · 2017-02-03 08:59:43

kirito wrote #303817:

I think thumbnails in separate folders would be good: it roughly halve the number of files per directory and would make it easy to implement multiple size thumbnails/images. That would be good to serve different size image for different devices/purposes.

You can do that already with the help of smd_thumbnail using an image profile to replace the regular txp thumbnail. That stores each image profile in a subdirectory named after the profile with the same filename. You’d need to move all the 123t.jpg files to thumb/123.jpg and so on. As you say, that halves the number of files in the directory (minus the folder).

But there is one possibly deal-breaking problem with this, depending on your setup. You need to use txp:smd_thumbnail instead of txp_thumbnail / txp:image thumbnail="1". If you are using the txp:article_image tag or some other tag-combo to show your images, it’s just a matter of adjusting your txp form to use the smd_ tag, and you’re ready to go. If, on the other hand, you have lots of manually inserted !/images/123t.jpg! textile comments in the txp:body, it’s a much bigger job…

jakob · 2017-02-03 09:16:48

Bloke wrote #303818:

There’s already a plugin for that, btw ;-)

Hehe, pipped to the post by a minute…

Perhaps viewing an image without one of these new ‘custom types’ will auto-create it, I don’t know. We’d need an upgrade to <txp:thumbnail> to allow you to specify a ‘type’ (default: type="thumbnail") but after that, everything should work fine and allow you to manage arbitrary thumbnails.

or add the attribute type="image-profile" to txp:image and txp:thumbnail just becomes an alias of txp:image type="main-thumbnail-profile".

Not sure. But I’d like to explore this idea as multiple image sizes is becoming a more common occurrence in the days of retina and high-definition displays.

Yes, retina images, but also different device sizes. Coming back to what you mentioned further above, it would be useful to be able to access the actual image dimensions of images in other image profiles, which you need for 500w etc. for srcset specification. Images don’t always match the respective designated dimension set in smd_thumbnail’s image profiles.

Also, please consider making the function extensible, so that things like focal point could be integrated as a plugin – that situation is currently a case where you need to make your own thumbnail images.

Bloke · 2017-02-03 10:01:51

jakob wrote #303821:

or add the attribute type="image-profile" to txp:image and txp:thumbnail just becomes an alias of txp:image type="main-thumbnail-profile".

That’s better. There’s a lot of duplication of functionality between image/thumbnail that I’d like to consolidate.

it would be useful to be able to access the actual image dimensions of images in other image profiles, which you need for 500w etc. for srcset specification.

I think we talked about this offline once, and I remember not quite “getting it” back then, despite your patient explanations. But some way to update our concept of how to manage and manipulate multiple images in the srcset world would be awesome.

To that end, if you have any insights in how we could arrange the UI, manipulate images on disk and allow the txp:tags to output the data in meaningful ways to suit responsive design, it’d be immensely valuable. I don’t really have much of a dealing with the sharp end of that concept, but if someone can define how things could be done better, we can build it that way.

Also, please consider making the function extensible, so that things like focal point could be integrated as a plugin.

Not heard of that (and the demo links are dead) but I like the idea of just using CSS to adjust an image’s centre point. Clever.

Textpattern CMS

Textpattern CMS support forum

#1 2017-02-02 17:01:48