images in yearly/monthly subdirectories?

tye · 2015-05-08 00:29:13

Hey – how’s things?

Is there a plugin, or way to organise images into yearly or monthly directories?

I’ve got a client with 10,000 images + and growing and my ftp only shows 5747 files – even using online ftp performance is… well you can’t really call it performance – its snail slow.

Bloke · 2015-05-08 09:26:32

tye wrote #290556:

Is there a plugin, or way to organise images into yearly or monthly directories?

Not that I’m aware of. This came up recently (though it actually turned out to not be the number of images after all in that case).

I wrote a tiny (as yet untested) plugin to essentially intercept and rewrite all image and file requests to farm them to date-based subdirectories. But it wouldn’t be a native solution at the moment as it requires two new callbacks to be added to core — one for images, one for files — plus a load of edits to the images and file code to cater for save/create/delete actions.

If it’d be of any use to you I could continue development.

Last edited by Bloke (2015-05-08 09:34:33)

colak · 2015-05-08 11:08:15

I think – although slightly more labourious – that this is possible if the date is hand written in the caption or the title field. Alternatively, you could make up image categories Something like, 012015, 022015…. 122015, etc.

Bloke · 2015-05-08 11:26:56

colak wrote #290568:

this is possible if the date is hand written in the caption or the title field.

Well if you wanted to organise them all by hand, sure. You could move your images according to any system you’d care to invent and do this kind of thing on the output side by constructing your URLs manually:

<txp:images>
   <img src="/images/<txp:image_date format="Ymd" />/<txp:image_info type="id, ext" />" />
</txp:images>

But unless I’ve misunderstood your post, it won’t help with the ongoing management via Txp’s interface because it still dumps all images in the main /images folder when you upload them.

If you bypassed Txp completely and uploaded all your files by hand, then you could do it, but you’d lose the ability to slice and dice them by category, use alt/caption metadata, etc. There needs to be a Txp layer (a plugin) that can redirect all read/create/save/delete operations to the correct real, physical location, plus core callbacks to allow that plugin to hook into the right parts of the UI workflow.

colak · 2015-05-08 11:45:15

Or it could be another image field(s) listing the upload/modify dates like in the articles tab. Then, all we will need is an additional tag to call that field in the selection and sorting of the images.

Having said that, it sounds very blogie and has nothing to do with the way a cms should work. In my view this is one of our problems as txp can both work as a cms (brilliantly) or as a blog (satisfactorily). Many of our users come here looking to create a blog for themselves and/or their clients but what txp can offer is much more, be that, a lesser blog functionality than a dedicated blog system.

Bloke · 2015-05-08 11:56:12

colak wrote #290570:

Or it could be another image field(s) listing the upload/modify dates like in the articles tab. Then, all we will need is an additional tag to call that field in the selection and sorting of the images.

I’m officially confused. Isn’t Tye’s question about physically storing the images in subdirectories to bypass file system limits and improve performance? The choice of storing them by date is entirely arbitrary, it’s just a mechanism to store fewer files in the root /images directory.

File systems start to groan/complain when the number of files (inodes) exceeds a few thousand in any directory. Storing images monthly seems like a reasonable way to do it. We already know the upload timestamp so /images/YYYY-MM/id.ext seems a good approach> it’s simple, repeatable, keeps the number of files per directory down, and also keeps the number of directories in the images folder small. With the rate of expansion growing at 12 directories per year, there’s a fair bit of expansion built in.

They could be stored by category name if it was preferred. Or hashed, or any system to distribute them more evenly. As long as it’s repeatable so the image can be stored in a location and always later be found by running the same algorithm (regardless of any meta data changes that might reasonably be altered), anything goes.

Last edited by Bloke (2015-05-08 11:58:17)

colak · 2015-05-08 12:29:08

Bloke wrote #290571:

I’m officially confused. Isn’t Tye’s question about physically storing the images in subdirectories to bypass file system limits and improve performance? The choice of storing them by date is entirely arbitrary, it’s just a mechanism to store fewer files in the root /images directory.

Whoops:) excuse the rave, that is indeed what Tye was asking! Sometimes the thoughts come faster than the eye.

ruud · 2015-05-08 21:07:38

Bloke wrote #290571:

File systems start to groan/complain when the number of files (inodes) exceeds a few thousand in any directory.

Some file systems, yes. EXT3 + dir_index or EXT4 can handle this just fine and I’m pretty sure most other modern file systems can cope with this as well (a client of mine has 6000+ files in the image dir). The FTP client not showing all images is a different problem. It also makes me wonder how the images are managed… via TXP or via FTP. It’s unusual to use both.

Bloke · 2015-05-09 11:45:15

ruud wrote #290587:

most other modern file systems can cope with this

Thanks for the info. Worth knowing, I sit corrected. As you say though, FTP is probably another thing entirely as it has to stream the directory index to the client and that can take a lot of resource. I’d be keen to know how files are managed in Tye’s setup too.

towndock · 2015-05-09 12:27:51

FYI, we now have almost 19,000 images on one site (and growing). I feared problems as this has grown over the years – but it has worked fine. We don’t attempt to FTP the image directory however – all just done simply through TXP.

tye · 2015-05-11 05:33:26

Whoops – didn’t get any notifications – sorry… (but I am subscribed??).

Isn’t Tye’s question about physically storing the images in subdirectories to bypass file system limits and improve performance? The choice of storing them by date is entirely arbitrary, it’s just a mechanism to store fewer files in the root /images directory.

Yes – it is :)

All the images are managed via the CMS – but I was using an old version of jbx_multiple_image_upload and some files got stuck in the tmp folder…. which I couldn’t get access too (I did eventually through web ftp)… I understand that isn’t a normal usage, and have replaced the plugin now anyway :)

After all the arguments too and fro… I’m not sure if it is needed or not, I work both in the CMS & via ftp, but I could be a one off case.

But if its not too difficult, it could be a great addition… a plugin would seem the best way forward if its possible

TXP is by far the best CMS for image handling I have worked with.

Textpattern CMS

Textpattern CMS support forum

#1 2015-05-08 00:29:13