Just Another Brick (in the) Wall

Over-engineering may be foolish, but under-engineering is just as perilous.

Advertisements

This month’s T-SQL Tuesday topic, thanks to Wayne Sheffield, is a doozy.  It’s difficult to admit failures, even if we eventually overcame them.  Mostly, it’s tough to fess up to those hours upon hours of what feels like wasted time preceding the hopefully inevitable breakthrough.

Let me tell you a tale of square pegs and round holes.

Get your mind out of the gutter.

And strap in.  This puppy went over 2k words.  I didn’t intend it that way, but, c’est la vie.

A Short History Lesson

I used to work for a software company that made K-12 educational assessment & reporting products.  A large part of K12ED is dealing with learning standards, these historically fraught named concepts that students are supposed to have “mastered” and learned during their time in the US public school system.  You ever hear a reference in pop-culture or media referring to “teaching to the test” or “state standardized testing” or similar?  That’s what they’re talking about.

In the late 90’s, a bunch of states, including California in 1997, finalized and centralized a “list” of sorts, almost a database (but not formally, because let’s face it, this was still the early days of what we know as modern IT), of said Standards, which came to be unofficially known as the “CA ’97 Standards”.  For over a decade, these learning goals dictated what was taught when in our schools.  Government funding incentives, based on state standardized test scores, drove instruction to the point that teachers had little if any time to dynamically mold their students’ minds into critically thinking, multi-faceted, creative individuals.

But this article isn’t about my bias.  This is about the broad, sweeping shift in the K12ED landscape called “Common Core”.  As the technology sector grew more mature and started permeating more “traditional” industries, a vocal minority of thought-leaders had what they deemed an epiphany.

Hey, what if we took the old-guard educational bureaucracy, and all those disparate state standards, and turned it into a unified, technology-driven learning ecosystem?

Sounds great in theory, right?  I mean, surely their intentions were good?  Well, you know what they say about the road to Hell and how it’s paved…

Gosh, there goes my bias again.  Sorry, let me just tuck that back in its pocket.

Anyway.  These new Core Standards introduced some new ways of thinking.  For example, that some learning concepts are inter-related and “cross-cutting” (a fancy way of saying that sometimes a Math concept requires a fundamental Reading-Literacy knowledge-point to fully grasp).  This had some very interesting impacts on the underlying technology systems which relied on, and housed, said Standards.  System which, I might add, had existed for over a decade at this point, in many cases.

Bringing it Back to Data

Our company’s system was one such.  Our key partner’s system, from which we quite literally inherited the traditional, relational database structure to house the CA ’97 Standards, was another.  You see, back before RESTful APIs ran the world, software systems relied heavily on what we call “local data stores”.  In order to show the teachers and administrators, who primarily used our system, the Standards, in which their students were performing well (or poorly), we had to relate those Standards to the test questions that said students were tested on month after month, year after year.  And, like so many other small businesses of the late 90’s / early 00’s, we had a trusty ol’ SQL Server lying around, just waiting to be loaded with all our precious data.

This was fine, for the most part.  The legacy Standards conformed reasonably well to a relational data model, even though we had to throw in a bit of hierarchy (using the good ol’ adjacency list scheme).  There wasn’t a complicated set of relationships from Standards in different subjects (Math, Science, History) to each other, and people didn’t care to do much in-depth analysis beyond “are my students getting well-prepared for the state tests?”.

Enter Common Core

You parents thought these things were complicated and convoluted — just Google “common core math problem” and you’ll find no shortage of critical satire.  Well, the underlying data structures required to store these new Standards were going to be significantly more complex as well.

One of my main jobs, for about a year or two, was to oversee the importation of said Core Standards into our SQL database system.  On the surface, it seemed reasonable — we had a hierarchy concept already, and we had a roll-up & drill-down mechanism for the handful of different “levels” of said hierarchy.  But it was all very static.  What that means, for us tech-heads, is that it was not easy to change or extend; not adaptable to new and expanded requirements.  The older Standards adhered to a fairly strict hierarchy, and each level of roll-up had a distinct name.  With Common Core, they broke out of that old mold, while simultaneously keeping some of the names only to change their meaning depending on context.

Think of it this way.  A group of cattle is called a herd.  A group of sheep is also called a herd.  And a group of seagulls is called a flock.

And I ra-a-an.. I ran so far a-wa-a-ay…

Sorry, where was I?  Right, group names.  So what if the government suddenly decided for us that a group of sheep will from now on be called a ‘gaggle’.  But only if they’re all female.  If the group contains any male sheep, it’s called a ‘herd’ still.  And groups of cattle will be still be called herds, unless it’s purely a group of bulls being raised for beef, in which case we call it a ‘meatsock’.

Have I lost you yet?  Of course I have!  This is pure nonsense, right?  Language does not work this way.  Moreover, hierarchies of groups of things do not work this way.

But I digress.  There was, despite my jest, a method to the madness of the new Common Core hierarchy groupings.  And I did learn it and understand it, for the most part.  The problem was that it threw our existing legacy data model to the wind.

Enter Academic Benchmarks

As usual with a legacy software system in a small company, the resources and buy-in for a data-layer overhaul were nil.  So it fell to us intrepid DB-Devs to shove that snowflake-shaped peg into the very square hole of the old relational model.  We sketched out plenty of ERDs, brainstormed how to tack-on to our existing structures, but nothing short of a miracle would make this new data conform to these old static norms.

Thankfully, the “geniuses” (and yes, that is used sarcastically) over at Academic Benchmarks, or AB for short (at least for the purposes of this post), had already done this.  And we paid them, thousands of dollars per year, for the convenience of using their GUIDs to identify Standards across different systems and vendors.  Never mind that they were just perpetuating the bad model of yesteryear; never mind that they provided zero support for data quality feedback loops.  We could happily take their Excel files or CSVs and load them nearly straight into our database.

Enter, and Exit, ASN

While I was searching for the words to express how insufficient our data model was, I came across this little gem from the Gates Foundation: Achievement Standards Network, or ASN.  (ASN stands for many other things, so as with all acronyms, it’s all about context; just fair warning for the Google-happy.)  The architects here had understood that learning standards needed a better and more flexible model, not only in terms of storage, but also in terms of data interchange format.  This new kid on the block called JSON had been making waves for a while, and was subsequently widely adopted by the tech industry in general, so it stood to reason that this would be the preferred format for publishing and serving the Standards from ASN.

Bonus: it was FREE.  Yes, really.  What a wonderful thought, I said to my team, to imagine never having to pay those crooks at AB another red cent!  Eventually.  After years of transition.  But alas, it was not to be.  See, AB had been around the industry for a long time.  They had their hooks in almost every learning content publisher and assessment vendor.  So as tempting as this shiny new source of academic truth may have been, sadly, it was not practical.

Enter the Contractor

Somewhere around the same time, we took on a promising new developer who, not only had a very strong background in Math and CS fundamentals, but who had also proven his worth with real world applications with actual “in the wild” deployments and users.  He was a bit arrogant, actually.  One could argue that he’d earned it, perhaps, but we didn’t appreciate always being told everything we were doing wrong, constantly, to the point that it was hard to hear the more important bits of wisdom and insight into how we could improve.

Despite that ego, one of his biggest contributions to the team was a fresh impetus to learn new things and branch out to new technologies.  To start looking at new Javascript frameworks.  To revisit OO fundamentals like Dependency Injection, and stop writing such procedural code.  To consider NoSQL data stores.  In particular, graph data stores.

Sadly, that last part came in too little, too late.

Side-note, on the crest of the micro-services wave, he diagrammed and proposed an entire system re-write for our core software product, using micro-services architecture and the concept of small purpose-dedicated data stores.  It looked real purty, but was ultimately and completely impractical for a company of our size.  If we were a “true startup” with millions in VC funding coming out the ears, and could afford to throw a brand new, young & hungry “2 pizza team” at the product, then sure.  But that was not reality.

The Brick Wall

No two bones about it: we had to support these new Standards.  So we soldiered on with our relational database tables.  And we tacked-on additional entities and relationships, logic and code, to make it “almost” do what the Common Core Standards wanted to do.  Effectively, what we were trying to do, was to shove that pretty round sparkly peg of graph data, into the dingy old square hole of a 15-year-old SQL table structure.

Somewhere along the way, AB caught up with the times and started offering other, less “flat” formats, for their Standards data.  So even though ASN was off the table, not all of our work toward JSON ingestion/conversion went to waste.  Consuming the data exports from the vendors wasn’t a problem — we’d already been doing this.  That was not the issue.

The issue, the brick wall against which we continually banged our heads, was the fact that we just plain couldn’t support the advanced & complex relationships and groupings (categorizations) of the new Standards.  Which turned out, in retrospect, not to be the end of the world, because honestly it would take years, if not decades, for the educational system’s old-guard mentality to even comprehend such relationships and categorizations, let alone how they could help shape classroom instruction toward better outcomes for their students.

Good lord, that sounded like a bunch of jargon.

What I mean, in plainer English, is that we spent a lot of time worrying and arguing about something that did not matter as much as we thought it did.  The consumers of our system, the teachers and principals and such, honestly didn’t care about all that.  They just wanted to know if their kids were on-track to be “Proficient” on the state tests so their funding would remain steady.  (I don’t give them enough credit, I know; teachers themselves also needed to know, on a regular basis, which kids needed special attention in what areas of learning, but that’s beyond the scope of most generalized reporting tools.)

Hindsight is Always 20/20

So what was the lesson here?  I don’t want to overlook the fact that we were still using the wrong data storage technology for the problem, fundamentally.  Or at least, the wrong data model.  But, we live in a real world where software needs to be delivered on-time, in-budget, with less than perfect teams who have less experience and expertise than they feel they should.  So instead of butting our heads against that brick wall, let’s try to remember to be pragmatic.  To be realistic about what’s feasible for who & when; and to adapt the parts and segments of our application infrastructure, appropriately and efficiently, to the business problems and use-cases at-hand.  Over-engineering may be foolish, but under-engineering is just as perilous.

How to Total Cars for Fun and Profit

Insurance is a wonderful thing…

This is a story.  And it’s longer than my usual post, so get comfy.  I may be stretching my own rules, but I swear, I’ll tie it back to databases… somehow! Let’s get started.

Back in December 2016, I was in an accident in my 2011 Mazda 3. Ironically, I was driving home from filling up with gas, plus I’d just had some maintenance done the month before. These things are ironic because the car was a total loss. “Totaled”, in layman’s terms. It means the damage was such that the insurance company would rather pay off the market value of the car, than pay for the repairs. Or, put another way, it means that the cost of repairs would be within nominal range of the vehicle’s value. Short version, I’m not getting the car back. Oh, and that gas fill-up and mechanic bill? Money down the drain.

dumping-money-down-the-toilet
I haz a bucket…

FYI, I was just fine – so the car and its safety-system did its job, protecting me from harm. Safety is an important part of choosing a car, kids… remember that.

safety-first

Anyway, the car gets towed off to a local yard, I have a day or so to collect my possessions from it, and then they tow it somewhere else to the “salvage yard”, where it becomes somebody else’s property and problem. Insurance sends me the check for the value of the car. I was actually worried that it wouldn’t be enough to cover the loan balance, because I was still making payments AND I’d refinanced midway thru the original loan term, lowering the interest rate and payment but also extending the term. But, thankfully, the car’s value was above the loan’s balance, so I was able to fully pay it off and still pocket some cash.

Unfortunately, I didn’t have “loss of use coverage” on the policy – meaning, no free rental car. So I pay for it, for a little while. Fortunately, around the same time, my parents were getting ready to dump their old 2000 Honda Accord for a newer one, so we started talking and they offered to us as a gift. (Did I ever mention how awesome my parents are? They are!) Excellent.

Actually, this is the same car that I used to drive around in high school, so it’s got some memories.

Including, of all things, my very first accident! You’ll see why this is ironic in a few paragraphs.

irony-alert
Someone call Ms. Morissette…

Now, this car is what we call a “beater”. It’s 17 years old, it’s been through the wringer, it’s got 190k miles on it; but hey, it’s a freakin’ Honda. It’ll last another 50k at least, if maintained properly. And it has been – faithful oil changes, scheduled maintenance and beyond. But it’s not the safest vehicle on the road; the e-brake light comes on sporadically, and the airbag warning light is always on, so we don’t actually know if the airbags (particularly the passenger side) will work. So we need to start shopping for a new vehicle, at least for the wife.

I won’t go into car-shopping here, it’s a pain the arse unless you use courtesy buying services like those offered thru your credit-union or your some kind of “club membership” or whatever. Long story short, we got a 2017 Hyundai Elantra, in black, not brand-new but used with only 3k miles on it (apparently they had buyer’s remorse).

black-hyundai-elantra
This is “Tigress”, or “Tigz” for short.

I will digress just for a second about black cars. They look pretty slick, even though they do show dirt a bit more than gray/silver (which is what the Mazda was). But you know what makes them look super-duper slick? Those “legacy” CA gold-on-black license plates. I convinced myself that I had to get those. Until I went to the DMV and found out that they’re $40 initially plus an extra $50/year on your registration fees. Jesus H… I get that they need to make money, but that’s ridiculous. Srsly. Ding the people that want those silly vanity plates, because I understand it adds a lot of processing/tracking overhead and makes the data (see? I told you I’d tie it back!) more complex. But this is the same old metal made by the same old prison inmates with a different coat of paint. Don’t pretend it actually costs anything extra for you to make them and pass them out. Anyway. Back to the story.

ca-plates-regular-vs-legacy
because it’s SOOO much harder to make black metal than white metal…

So we get the car, the wife’s driving it home. She’s heat-sensitive, and it’s been a pretty long, warm day. She ends up passing out for a second and veering off the road to the right shoulder, which is a small dirt embankment into some bushes and trees. Fortunately enough, she realizes what is happening and she’s able to bring the car to a gentle stop without actually hitting anything. So the car’s only real damage is some scrapes & bruises on the front end, a bit of scratching on the sides, and some dings to the under-carriage-panel. Now, me being the savvy consumer that I am, I’ve already added it to our insurance policy and added rental coverage. The new car goes into the shop before we’ve even had it a day, but we get a free rental while it’s there. (Ford Fusion – I like it alright, but the wife hates it; she has a bit of an anti-Ford bias.) Insurance covers about 1.5k damage for the bodywork, it gets done, we get it back in less than 2 weeks. Yay.

Alright, here’s where it gets fun interesting.

oh-it-gets-better
I don’t know why, but this is one of my favorite lines of his from this movie.

That was all back in December/January.

March rolls around, and I’m driving the Honda home from work. There’s a sudden pile-up of stopping cars in my lane and I can’t stop in time, so I run into the SUV in front of me. Fairly low speed, nobody panics, we pull over and start exchanging info and pictures. Now, my bumper is nearly detached, and my hood is quite scrunched in at the point of impact. This is because I hit his tow hitch, which stuck out from the rest of his rear body quite a bit. So even though he literally has a 1-inch scratch on his bumper, I’m looking at significant damage. But it seems mostly superficial, so I figure, well, I might not even need insurance, and he certainly doesn’t care enough to report it unless I do, so he leaves it up to me.

He helps me rope-up the bumper so it doesn’t fall off (he was such a nice guy, no joke!), and I start driving the rest of the way home. Quite a distance, mind you (I have a 60 mile total commute). After a little while, I start seeing smoke coming from under the hood. Fortunately it’s white, not black, so I know I’m not in terribly immediate danger. But I pull off to a gas station and take a look. Well, I can’t actually open the hood due to the scrunchy-ness, but I peer inside and see that there’s a significant bit of frame damage, and the radiator looks hurt. Sure enough, it would turn out that that was the biggest problem – the radiator (and compressor) would need to be replaced, and the frame around it needed repairing/re-welding.

old-car-on-fire
Not quite…

This is not a small job. I take it to a body shop first, but as they look inside and see what I saw, they know that it’s beyond their scope, so they send me next-door to a full-service mechanic & repair shop. Next day, he gives me the estimate: $2.7k. Now, about this time, I’m talking with the insurance reps. I know they’re going to want to total this car – it’s KBB value is literally just over $2k, and these repairs are significantly more than that. What I was trying to ask them, and never got a straight answer, was whether we could file the claim for a lower amount, by asking for the mechanical repairs only. Remember, this car is a “beater”. We don’t really care how it looks, we just need it to run. And the shop was kind enough to provide that “bare-bones” estimate as well – only about $750.

honda-wreck-damage-picture
You can see where the tow-hitch rammed thru the bumper/grill in towards the frame; the blue arrow points to the frame inside that got the brunt of it.

But then my insurance adjuster did two things that were very insightful & much appreciated.

I have to give a shout-out to Safeco here, because throughout all of this, they’ve been immensely helpful and easy to deal with. (Even though I mentioned not answering my question in the paragraph above, as you’ll see, that was really my own fault for not understanding the process, and it was a moot point anyway!) So if you’re in the market for a new insurance policy, definitely check ‘em out.

First, because this shop was not an official “authorized partner”, she couldn’t accept their estimates as gospel; but, she could offer this newer “pilot” program whereby any shop (or even the customer) could submit pictures of the vehicle and the damage, and, provided enough detail and the right angles, a 3rd party estimator could assess the damage and estimate the cost. Great!

Second, she heard me out as I explained the concern with totaling the car, and understood that I really wanted to keep the car after simply getting it mechanically sound. But, she clarified, because I had collision coverage on this car, they (the insurance company) literally “owed me” the full cost of those repairs or the vehicle value, whichever is lower. So in fact, I would be doing myself a disservice and actually losing money if I tried to simply file the claim for the lower “bare-bones” amount, just to avoid the total-loss.

Instead, she explained, what you can do is keep the vehicle, even after it’s been declared “totaled“.

There’s a process and paperwork to this, and it involves the DMV, obviously. But because the insurance policy will still pay me the value of the vehicle, I should have more than enough to get the minimum repairs done and pocket the rest. Yay!

smiley-with-money
cha-ching!

Now, the process. The CA DMV has done a fairly decent job of documenting this, but it’s still unclear (at least to me) what the order of operations is. There are 5 things you need:

  1. Salvage title (which is different than the regular title, aka pinkslip)
    • DMV form REG 343, which you fill out yourself
  2. Salvage certificate
    • REG 488c, which you also fill out yourself
  3. Owner retention of salvage vehicle
    • REG 481, which your insurance company completes & sends to the DMV
  4. Brake & light inspection (to make sure it meets road safety standards)
    • Certificates are printed & given to you by the inspecting shop
  5. Full vehicle inspection (again, safety & compliance)
    • REG 31, which is completed by DMV personnel only

Number 4 can be done by many authorized 3rd-party shops, most of which also do smog tests and such things, so they’re not hard to find. The rest are DMV forms, as noted above. (#5 can be done either by the DMV or by CHP; but, CHP has quite a narrow list of “accepted” vehicles which they’ll inspect for this purpose, and honestly their appointment “system” for trying to get them done is horrendous, so it’s easiest to let the DMV do it.) But again, what’s the order in which I should do these things? Well, let me tell you!

joker-just-let-me-finish
I’m trying!

First, you get that payout check from your insurance, and you get the repairs done. Then you take the car to a brake/light inspection place (#4 above), and get that “certificate” (much like a smog certificate, it’s an “official” record that says your vehicle passed this test). Actually, if the vehicle hasn’t been smog-checked recently, you probably need that too. Mine was just done in 2016 so it wasn’t necessary.

Ooh! Another database tie-in. Okay, we all know a car’s VIN is like the primary key of the DMV’s vehicle database, right? Plate#s you can change, but the VIN is etched in stone steel. But they’re largely sequential – so two 2000 Honda Accords are going to have mostly the same characters in their VINs, up to the last, say, 2-6 numbers (ish.. I’m nowhere near knowledgeable enough about the system, I’m just guessing based on my observation of what happened to me). So when the paperwork comes back from the insurance, it ends up with the wrong VIN, off by 3 #s at the end. But I don’t realize this until I check with the DMV as I’m filing the accident report. Also, you can use online services to look up a VIN and find the basic info about it, but again, because I had such similar VINs (my correct one, and the insurance’s one from the papers), both turned up the same descriptions, down to the body style and trim level (4 door sedan, LX, if you’re curious). The only way we actually found the mistake was that the DMV was looking up “ownership” info based on the VIN, and when the agent read me the name on file, I was like “whodat?”, since it wasn’t me or my father, and then I went back to my pinkslip and checked it there, as well as on the car’s door-panel.

The lesson here is, always double-check your VIN when filing paperwork, especially with the DMV. Moving on.

vin-number-atm-machine
STOP adding redundant words to acronyms phrases!

Before you go further, you need to actually make sure that the insurance and/or the salvage yard has officially notified the DMV of the vehicle being a “total loss”. (See #3 in the list.) In my case, they hadn’t – it had only been a month (between the actual payout and the first time I went to the DMV). So I have to check again before I go back.

But, since I was there, I made the DMV agent answer all my questions and specify exactly what I needed to do to complete this process, and the order in which to do it. Which is why I’m now writing this and sharing with you!

Once that notification is done, the DMV will have record of the vehicle being a “total loss”, or “salvage”. Then you can make a new DMV appointment, go in, and get #5 and #1-2 done all at the same time, in that order. I.e., go to the “inspection” or “inspector” side first, have them do the inspection and fill out the form (REG 31). Then go to the appointment line and take all your paperwork to the agent that calls you. So that’s your “inspection-passed” form (REG 31), your salvage title form (REG 343), your salvage certificate form (REG 488c), and your brake & light certificates. If you have a copy of the insurance co’s REG 481, might as well bring that too! You also need your license plates – you have to “surrender” them, which means turn them in and get new ones (not that same day, obviously – I think they still mail them to the DMV and you have to go pick them up… but I’ll find out soon).

austin-powers-dmv-live-dangerously
Life on the edge, man!

Finally, to add a little icing on this crap-cake. I was driving the Hyundai to work, literally the next day, and I got rear-ended by another driver who wasn’t paying attention at a red-light. Again, super low speed, minor damage, but, another visit to the body shop for that poor black Elantra, and another week with a rental car. (Hyundai Santa Fe this time, which is actually quite nice, and if we need a small/mid SUV in the future, I’d definitely consider it; but due to my commute, we swapped for another Ford Fusion, this time the hybrid model, which again, I enjoyed, but the wife did not. Hey, you win some, you lose some.)

crap-cake-smiley
Frankly, 95% of Google image search results for “crap cake” were gross and offensive, but this one was almost cute.

So that’s the story of how we totaled two cars (and damaged one car twice) in less than 4 months.

And that’s the reason I’m now taking a van-pool at least 2 days a week.

I’d always been a fairly safe & cautious driver, but I’ll admit, this long commute had turned me into a bit of a road-rager. Impatient would be the polite term. After all this, I’m back to my old cautious slow & steady ways… for the most part. I still get little flashes of panic when I go by the intersection where the Mazda wreck happened, and I’m always reminding the wife to stay cool and drink her water. She’s never had that happen before, and never felt like it since, so I’m sure it was a one-time fluke, but still.. the DMV wants her to re-test to get her license back, even after her doctors cleared her to drive. That’s a whole other topic, for another time. I will note that none of these incidents were due to cell-phone use, so at least we’re not guilty of that particular vice.

keep-calm-and-drive-safely

Thanks for reading!

Now, go out there and DRIVE SAFE.