Go to main content

Textpattern CMS support forum

You are not logged in. Register | Login | Help

#13 2019-10-10 18:40:37

Bloke
Developer
From: Leeds, UK
Registered: 2006-01-29
Posts: 11,250
Website GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

gaekwad wrote #319622:

However, the dev site script, which uses the same routine as the release and beta sites, inserts the Publisher…and then seems to stall.

Uh oh, your suggested optimisations appear to have backfired, Pete :p

Seriously, I dunno why it would stall like that. I can understand that PASSWORD_DEFAULT (which uses BCRYPT) might take a bit more processing power to generate than what we were using before but I would have thought the PHP team would optimise it if they could.

In addition to what Oleg suggests, does it stall or redline if you generate just a handful, say change the loop to $i <= 10? Maybe the sheer quantity is making the CPU melt as it does the bcrypt calculations and we need to batch them up or something.

Very weird.


The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.

Txp Builders – finely-crafted code, design and Txp

Online

#14 2019-10-10 18:46:41

Bloke
Developer
From: Leeds, UK
Registered: 2006-01-29
Posts: 11,250
Website GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

The only other thing I can think of is that we don’t have enough space in our pass field for the latest version of PHP’s implementation (although I’d expect it to truncate if so, not burn up).

One of the things the PHP docs do state is that 60 chars (as of PHP5.5) is enough, but the idea is that they’ll gradually increase the size of the computed hash with each version of PHP and suggest 255 chars as a suitable max. Maybe PHP 7.4 has already breached our 128 VARCHAR limit?

Can’t see that being the issue but I’m at a bit of a loss as to why it’s behaving like this in 4.8.0-dev.


The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.

Txp Builders – finely-crafted code, design and Txp

Online

#15 2019-10-10 18:58:30

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

etc wrote #319625:

As suggested by @petecooper,

Ahem.

Can you try whether echo password_hash('hello', PASSWORD_DEFAULT); is working?

It works. Refreshing the page provides a different hash each time.

Offline

#16 2019-10-10 19:01:50

etc
Developer
Registered: 2010-11-11
Posts: 5,028
Website GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

gaekwad wrote #319628:

It works. Refreshing the page provides a different hash each time.

Fast and less than 128 chars?

Offline

#17 2019-10-10 19:08:04

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

Bloke wrote #319626:

In addition to what Oleg suggests, does it stall or redline if you generate just a handful, say change the loop to $i <= 10? Maybe the sheer quantity is making the CPU melt as it does the bcrypt calculations and we need to batch them up or something.

Bingo. Good spot. I ran time against a different number of accounts:

10 accounts

real	0m0.822s
user	0m0.767s
sys	0m0.004s

100 accounts

real	0m6.971s
user	0m6.519s
sys	0m0.020s

…so I assume that adding a thousand accounts a) takes roughly ten times as long and b) chews up enough CPU & RAM on a low-grade VPS that it effectively stalls.

Maybe the sheer quantity is making the CPU melt as it does the bcrypt calculations and we need to batch them up or something.

I’ve flipped back to the old squirt-the-SQL method for 4.8.0-dev on the new server, that works A-OK, and I’ll divvy up the workload to get the build duration sorted. I have one of those plans up my sleeve that might be needlessly complex but will mean things are wicked fast at restore time. To be continued!

Regardless of my complexity, thank you both very much for your advice and input – greatly appreciated.

Offline

#18 2019-10-10 19:13:12

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

etc wrote #319629:

Fast and less than 128 chars?

Yes and yes.

dev-demo.textpattern.co/add_users_etc_test.php (live until 2100UTC),

Offline

#19 2019-10-10 19:17:02

Bloke
Developer
From: Leeds, UK
Registered: 2006-01-29
Posts: 11,250
Website GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

gaekwad wrote #319630:

10 accounts real 0m0.822s... / 100 accounts real 0m6.971s...

Jeez, didn’t think it’d be that pronounced. Wow. I wonder if the opcode behind the scenes is doing some background stuff (which makes hardcore pipelining impossible) to try and improve the perceived speed of these intensive calculations, which works great for one, but trips over itself if you try and batch ‘em up.

Veeeery interesting.


The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.

Txp Builders – finely-crafted code, design and Txp

Online

#20 2019-10-10 19:19:03

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

Bloke wrote #319632:

I wonder if the opcode behind the scenes is doing some background stuff (which makes hardcore pipelining impossible) to try and improve the perceived speed of these intensive calculations, which works great for one, but trips over itself if you try and batch ‘em up.

I can get you a login if you’re curious…everything on the server is disposable and rebuildable, so little to no risk.

Offline

#21 2019-10-10 19:23:44

Bloke
Developer
From: Leeds, UK
Registered: 2006-01-29
Posts: 11,250
Website GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

Actually, looking again at the numbers, I guess they do ramp up linearly(ish), and it’s not the sort of thing they can exactly take a shortcut over, seeing as it’s a cryptographic hash and needs to be as random as possible.

Guess it’s one of those things we have to live with as the complexity increases to stay abreast of hardware improvements and server farms crunching data. So, uhh, yeah, gonna need to do some optimsations our end to make it viable.

Looking forward to seeing what you come up with but if you need a hand with anything, give one of us a shout.


The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.

Txp Builders – finely-crafted code, design and Txp

Online

#22 2019-10-10 19:33:17

etc
Developer
Registered: 2010-11-11
Posts: 5,028
Website GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

Bloke wrote #319632:

Veeeery interesting.

It looks like that’s by design.

Offline

#23 2019-10-10 19:39:15

Bloke
Developer
From: Leeds, UK
Registered: 2006-01-29
Posts: 11,250
Website GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

etc wrote #319635:

It looks like that’s by design.

Yeah, I figured that after posting my ill-informed reply!

For the demo server, it could be serviceable to reduce the default $cost since the accounts only last 3 hours anyway so anyone hacking them to produce rainbow tables or whatever will be wasting their time.

However, we don’t exposes the cost in our API so the hashing algorithm is stuck at whatever the PHP team deem is acceptable for any given release. Is the cost something we might like to expose so Pete can use core functions here instead of having to generate them directly in raw PHP and populate the DB by hand?

EDIT: not sure exactly how we’d do it though for best.

Last edited by Bloke (2019-10-10 19:40:39)


The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.

Txp Builders – finely-crafted code, design and Txp

Online

#24 2019-10-10 20:19:01

gaekwad
Server grease monkey
From: People's Republic of Cornwall
Registered: 2005-11-19
Posts: 4,134
GitHub

Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?

Here’s one of my overly-complex plans…

Assuming 3 hours (10800 seconds) between rebuilds, there’s ample time to hash 1 password per second, or hash a password, wait one second, hash the next, and so on. So, given that a server can tick along hashing these passwords in the background, I can pipe them into a headless Textpattern database, and as part of the preflight plan as the rebuild approaches, export the headless Textpattern database to a file…then just squirt that into the live Textpattern in less than a second.

The passwords don’t change, at least not right now, so presumably that headless database that took three hours to ferment can just be squirted in every three hours, right? Do the work once, splat it every three hours. Fewer CPU cycles, so marginally lower power usage.

Like a PHP microbrewery…

Offline

Board footer

Powered by FluxBB