Textpattern CMS support forum
You are not logged in. Register | Login | Help
- Topics: Active | Unanswered
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
Bloke wrote #319626:
In addition to what Oleg suggests, does it stall or redline if you generate just a handful, say change the loop to
$i <= 10? Maybe the sheer quantity is making the CPU melt as it does the bcrypt calculations and we need to batch them up or something.
Bingo. Good spot. I ran time against a different number of accounts:
10 accounts
real 0m0.822s
user 0m0.767s
sys 0m0.004s100 accounts
real 0m6.971s
user 0m6.519s
sys 0m0.020s…so I assume that adding a thousand accounts a) takes roughly ten times as long and b) chews up enough CPU & RAM on a low-grade VPS that it effectively stalls.
Maybe the sheer quantity is making the CPU melt as it does the bcrypt calculations and we need to batch them up or something.
I’ve flipped back to the old squirt-the-SQL method for 4.8.0-dev on the new server, that works A-OK, and I’ll divvy up the workload to get the build duration sorted. I have one of those plans up my sleeve that might be needlessly complex but will mean things are wicked fast at restore time. To be continued!
Regardless of my complexity, thank you both very much for your advice and input – greatly appreciated.
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
etc wrote #319629:
Fast and less than 128 chars?
Yes and yes.
dev-demo.textpattern.co/add_users_etc_test.php (live until 2100UTC),
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
gaekwad wrote #319630:
10 accounts
real 0m0.822s.../ 100 accountsreal 0m6.971s...
Jeez, didn’t think it’d be that pronounced. Wow. I wonder if the opcode behind the scenes is doing some background stuff (which makes hardcore pipelining impossible) to try and improve the perceived speed of these intensive calculations, which works great for one, but trips over itself if you try and batch ‘em up.
Veeeery interesting.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
Bloke wrote #319632:
I wonder if the opcode behind the scenes is doing some background stuff (which makes hardcore pipelining impossible) to try and improve the perceived speed of these intensive calculations, which works great for one, but trips over itself if you try and batch ‘em up.
I can get you a login if you’re curious…everything on the server is disposable and rebuildable, so little to no risk.
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
Actually, looking again at the numbers, I guess they do ramp up linearly(ish), and it’s not the sort of thing they can exactly take a shortcut over, seeing as it’s a cryptographic hash and needs to be as random as possible.
Guess it’s one of those things we have to live with as the complexity increases to stay abreast of hardware improvements and server farms crunching data. So, uhh, yeah, gonna need to do some optimsations our end to make it viable.
Looking forward to seeing what you come up with but if you need a hand with anything, give one of us a shout.
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Offline
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
etc wrote #319635:
It looks like that’s by design.
Yeah, I figured that after posting my ill-informed reply!
For the demo server, it could be serviceable to reduce the default $cost since the accounts only last 3 hours anyway so anyone hacking them to produce rainbow tables or whatever will be wasting their time.
However, we don’t exposes the cost in our API so the hashing algorithm is stuck at whatever the PHP team deem is acceptable for any given release. Is the cost something we might like to expose so Pete can use core functions here instead of having to generate them directly in raw PHP and populate the DB by hand?
EDIT: not sure exactly how we’d do it though for best.
Last edited by Bloke (2019-10-10 19:40:39)
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
Here’s one of my overly-complex plans…
Assuming 3 hours (10800 seconds) between rebuilds, there’s ample time to hash 1 password per second, or hash a password, wait one second, hash the next, and so on. So, given that a server can tick along hashing these passwords in the background, I can pipe them into a headless Textpattern database, and as part of the preflight plan as the rebuild approaches, export the headless Textpattern database to a file…then just squirt that into the live Textpattern in less than a second.
The passwords don’t change, at least not right now, so presumably that headless database that took three hours to ferment can just be squirted in every three hours, right? Do the work once, splat it every three hours. Fewer CPU cycles, so marginally lower power usage.
Like a PHP microbrewery…
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
That’s crazy enough to work. Love it!
The smd plugin menagerie — for when you need one more gribble of power from Textpattern. Bleeding-edge code available on GitHub.
Hire Txp Builders – finely-crafted code, design and Txp
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
Some (crude) tests:
for ($i = 12000; $i <= 12099; $i++) {
$name = 'managing-editor' . $i;
create_user($name, "{$name}@example.com", $name, "Managing Editor #{$i}", 2);
sleep(1);
}CPU load ticks along at 8-12% (~10% average), real: 1m47.767s, user: 0m6.747s, sys:0m0.040s.
Then, as above with a 2 second sleep…
CPU load ticks along 2-12% in a up-down zigzag motion, real: 3m28.037s, user: 0m6.762s, sys: 0m0.028s.
…and just for fun, as above with a 0 second sleep…
CPU pegged at 100%, real: 0m6.905s, user: 0m6.505s, sys: 0m0.032s.
I’ll experiment with usleep() and see if I can find a balance between performance and time taken.
Visually, three tests. Leftmost is 1 second, middle is 2 seconds and rightmost is 0 seconds.
Offline
Re: 4.7.3 - any changes required for MySQL 8.0 compatibility?
And going wildly off-topic, if I’m generating users and passwords that slowly and I have CPU cycles to spare, I’m tempted to compile a list of hundreds of first names, initials and last names to give each user a ‘real’ name instead of “Managing Editor #437”.
Offline