Continuing to feel bleh. Body is shot. Oi. Need more sleep.
So when this actually posts, I’ll be asleep. Magic you say? Yes. Super Secret Magic.
I’ve called it a night, and called it a week, called it done. I’ve been running running running, and when you have someone telling you they are going to help “fix your life” and yet you’ve been running on fumes because of their projects? Time to take a step back, get rest, and avoid the stupid conflict of wanting to slap a customer.
Thus I am writing: Goodnight.
So about a month or so ago I picked up a gig on a software project. I talked about it recently in this post. I mentioned a file issue with how things were inserted yada yada yada. Well been working on it tonight let me give you a guide – in this image that’s about to follow, G = GOOD, image is in the database, valid, active, 100% working, B = BAD , should not be on the server, should be deleted. AKA: Using my good storage!
So you’ll note Good = 17,126 (and it stays there) however B= 120k and growing. That’s a real time comparison between the file structure and the database for uploaded images. Oi. The entire thing just ended and this is still my dry-run but the dry run reports:
Good Records: 163,038
Bad Records: 2,270,934
That’s 2 MILLION files in the structure which have no corresponding record in the database, thus are NOT used for anything. ANYTHING. Now to adjust the app to not just scan, but delete.
Update: Already freed up 60GB of storage, gone through sections, no missing images. We may have a free hard drive soon! Woohoo!
Update 2: Really need to goto sleep exhausted, but been doing this folder by folder, and on the 9’s which I may leave till morning as it over 70GB free’d from garbage that didn’t belong. Woot!
So Lilly last night didn’t sleep much. I think when she started to feel better and meds helped congestion she was like:
Sadly, that had not been my goal, but we partied none the less (and by partied, I mean of course sat in bed with me saying “Do you want to goto sleep?”) at one point since I had work, she sat reading books to herself, and was having a blast.
Tonight so far has bee mixed, we crashed out early, then she woke up, a bit ago, hopefully, she’ll crash out soon, but as awake as she seems? Maybe 1.
Update: Yup. 12:55, zonked out. Now to work! I have 100GB+ of data to remove without hurting what’s in the database! (ugh)
A while back in an effort to try and make my life easier honestly, I offered a majority of my client base “Managed WordPress” it’s great software but it’s also easy to forget, it’s software.
Like all good software, it updates, and so do things like Plugins and Themes. So I grabbed some software as an investment to help make things easier, however customers don’t always take advantage of it, and too branding it to 1 singular customer experience (Aka Not to resellers, server customers etc.) the goal is simple – make it portable, make it a value add or even more so a marketable option.
So this past week/weekend (it’s all a blur) I grabbed a new domain, parked the software on it, and over the next few weeks going to pitch it to server customers as a monthly/yearly product (thinking yearly) the goal is to help them save money in the long run, as if they can update their WordPress installs, I don’t spend hours repairing, and while I don’t get then a paycheck for repairing, I don’t also get stuck in front of a terminal cleaning up WordPress! More time to spend elsewhere.
Doing some basic clean-up tonight, been doing a lot of server work lately, so far all running smoothly, but still more to do this week. So going to try and establish a 1AM bedtime, 2AM at the ABSOLUTE latest, a lot of daily routines I want to build in more, some writing time, as it cools some drone time (for the youtube channel!) and hopefully before month’s end a regular afternoon/evening walk! Want to get my steps up, and some of this gut gone!
Hope everyone has a great week!
While others don’t labor, I get to. The joys of the holiday’s folks get off, they tend to look in on their websites. So here we go, a busy day will come. I’m going to go to sleep and rest for a bit before it starts.
Today was a day of rest. Not much went on. Till tomorrow.
A lot of things to do lately in the realm of business. Seems like it never stops. I have a project for a popular magazine. Not going to stay what it handles, who it’s for – but let’s just say it hired a programmer once upon a time, and that programmer made a HUGE mess in which nearly 18GB (in a single folder) is currently sitting on server, with no-easy-way-to-clean-it-out.
Now I recently inherited this issue, and it’s a doozy! Now there are like 200 different ways while napping I can think of to organize images. Not claiming to be the smartest man on the planet, but let’s just go over how this is done.
First information comes into the magazine, a bulk amount of data is imported, each article has image assets so let’s say article number #1492 has 5 images attached to it to gather and store-on-server, perfectly fine. The import gives me those images in an image that looks like this:
Now I have no control over naming conventions so of course I too would take and find a way to keep track of images, so maybe a unique ID like oh I don’t know the article ID + filename so something like:
This way should I ever need to remove images I can script out just about ANYTHING to take that article number and purge the file system of it should I desire to remove the article, no assets left over, life is happy, all is good! But sadly in this case each night the data file comes back in, same image names attached to something I didn’t want anymore, including some articles which expire by default, so assets can get mucky VERY quickly just by nature.
The oddity is the guy who programmed did something just plain odd by how my thinking works – as he inputted the assets into the database when MySQL inserted it to the uploads for media to track, he appended the ID of the image, not the ID of the article. Fine right? Wrong. No checking.
So now a random image ID generated by it’s place in line has literally (and this isn’t exaggerating) about 125 entries in the file system, for a single file, oh but wait it creates a print, thumbnail versions too – so 375 images (all the same file) – and mind you if an article has say 5-10 images – I just laughed so hard I pee’d a little. Even on the short end if only 5 images (which many have more) we’d be looking at 1,875 images. Duplicated. Un-needed.
Oh, there’s more. It’s spread out across an INSANE file structure, so let’s say the image ID was 4985 like I noted above, it would upload that file & it’s adjusted brothers and sisters to the following folder in the structure:
4985 = uploads/4/9/8/4985-f83r5713209sdf9834.jpg
So these duplicates are spread out across a HUGE file structure, now thankfully I know I can delete a TON because the current picture ID’s short of a few 190’s is in the 200k’s so just the 0’s and 1’s? 18.2GB. Wasted space. I did a check of the recent imports? So many duplicates and the entire image folder of uploads/9/ is 47GB in storage. Top that off with one of the last imported? Has references back to the 1’s folder! 03d2bedbe4c84229b5263785d61f432c is the filename (.jpg) but it has references back when it’s image ID was 121! Oi!
The big chore now is writing a tool which will cross reference everything in the file structure against the DB, really clear out the garbage and get things running smoothly again. I have a meeting with the guy who tossed this project in my lap, while I love hours solving these stupid problems, I too also like sleep, low stress, and not having the file system fill up with so much garbage it runs out of storage and breaks the data import in the AM, I am tired of problems, I want happy solutions.
So busy. I need sleep.
Examining 358356 files, 111 dirs (in 1 specified)
209598 duplicate files (in 88853 sets), occupying 10715 MB (aka 10.715GB)
Note this is the number 1 folder. That’s not including 0,2-9 (as well as some test folders for what may have been meant to be article ID’s)
Examining 2317950 files, 3157 dirs (in 1 specified)
2104868 duplicate files (in 182205 sets), occupying 119834 MB (aka 119.834GB)
All folders scanned. Horrific waste of space. This guy was being paid an arm and leg to get this code up to snuff, and all he would have had to do was make the prefix the article ID and he’d of had an insanely EASY way to prevent duplicates, even if he overwrote it each time, no duplicates, always would = article ID first. Oi. Now the fun task, can I re-write themes & such to match the article ID prefix? or do I go with my “Let me re-build this on top of WordPress?” idea. Hrmm. I’ll sleep on it.
I’m exhausted. I have a large pile of things to do, and continually less energy to do it. I’ve submitted my hours for the week, amazing what you can get done from the hours of 11pm -> 6am, but does not do anything good for your body when a 2yr old wakes up at 8am and is ready to go!
I just ate a small snack of carrots, waiting on a server to re-compile, and watching a network test continue to unfold and see if my internet is sure’d up now. But been doing a lot of little cleanups here and there, new software I grabbed for backing up sites is doing an amazing job, going to deploy it on a few customer projects too.
This week accomplished some things, but overall? I’m tired still. I really just want to veg out this weekend if at all possible, but such is life, no rest. 🙁
May swap to the bed and put my feet up for a bit, still at least an hour to go on this project with the re-compile, not sure when I’ll get to sleep.
I’ve got not time. I need to find more time, just never is enough. Tired, lots to do, too many fires derail, finishing projects deadlines approaching.