Double Check

Some lessons are learned the hard way.  Fortunately, some are only learned the almost-hard way.  This is a story about the latter.

I was working on a user profile feature.  A neat little slide out div that allows a logged in user to edit their name and email, and reset their password.  As with most web development projects, it was a bit of an iceberg.  Creating the slider and simple form was fairly easy, but also on the list of necessary items was: Javascript validation and visual feedback, PHP processing of submitted info on the server side and server side validation of that info, and SQL statements for updating the database.  The last two are where I made my almost-hard lesson.

On a couple different recent projects, we’ve used a nice ORM system.  Short for object relational mapping, an ORM basically prevents you from ever having to write SQL directly.  Django uses one- you just create classes in your models.py file to ‘model’ your data.  .NET also does (I think)- add a new .cs file to your models folder and you can create the table with get and set methods, type tagging, etc.  Those frameworks will then produce the SQL commands (specific to the type of database you’re using) and you’re good to go.

But on this project, it’s an older setup – just a simple PHP/MYSQL system.  I’m generally ok writing SQL, but when you get out of practice, you make stupid mistakes.  I grabbed the right table, used an UPDATE command on the row denoted by the client id, and tested.  What was missing?  Well, in this system, there are two important keys for users: the client_id (basically a group- one client can have many users) and the user_id (which is the actual unique id for a specific user).  By only grabbing the client id, I’d created a method to update all users related to a client at once (not exactly a useful function).

I tested the update and saw my user’s profile save the correct info- great!  But then I checked the rest of the users for that test client- they were all named ‘Testy Guy’ and had email ‘test@test.com’.  After a few moments of panic, I realized that my mistake, while really stupid and dangerous, was not as bad as I’d thought.  It had only updated users for that one test client (I had, after all, grabbed the client_id) and I was still working in a full test environment (testing only database and code base).

So, an almost-bad mistake.  Still, a really careless mistake.  But a reminder and good lesson to always check my work.  In this case, adding an important client side check:

  • Checking that the unique username is not null or a blank string when submitting (this will ensure that the server code gets all the info it needs)

And an even more important couple server side checks:

  • Check that username again, return false with an error if blank
  • And use the mysqli_num_rows function to ensure only one row (and therefore, user) is being updated.  If that function gets more or less than one, return false with an error.

And it renewed my interest in getting into automated testing.  I know- I’m a monster for not doing this sooner, but it hasn’t been pushed at any job I’ve worked so far.  And it’s not really a popular course/tutorial topic.  There’s a rush (I know I’ve felt it) to get something working and up on the screen- but the more I read about (and experiment with) truly testing code, the more I realize that it will pay off in the long run.  More on that next week!

Don’t Repeat Yourself

DRY- of course.  One of the core principles (so I’ve read) of software development.  Repeating yourself in your code or design almost always leads to trouble- repeated bugs, fixing/editing things in multiple locations, general confusion, dogs and cats living together- pretty much all the bad parts of the Bible.

But sometimes I forget that those little machines running behind the scenes don’t really play by the rules.  Like the PHP parser.  That thing (or whoever wrote it) is way smarter than I am.  But one of its main functions is often to repeat itself (the project we’re working on makes liberal use of while loops to output content stored in the database).  So, while this program is happily doing its job, looping over rows and printing titles and text fields, repeating itself, I’m banging my head on the keyboard trying to figure out why my little jQuery ui widget is grabbing 3 elements each call instead of the one it should.

And therein lies the moral of this short Friday night story.  Throwing your css and js code into your php file while you’re developing seems like a great idea- that way, you can just work in one file, and break it out into the proper modules when everything looks right.  But be careful where, within that php file, you put your js.  Because if you accidentally put it inside one of those awesome php while loops, you’re going to end up with multiple identical functions.  And you’re gonna have a bad time.

Seriously- pretty soon I’m going to have to rename this thing to “I’m not as dumb as my code looks.”

Comma Separated Valuable Data

I’m sure there are some people who love the process of monthly reporting.  I’m not one of them.

I get it- they’re important.  Statistics, data, information about information- it’s all useful stuff.  The stuff that makes a business run.  But there’s something about staring at a spreadsheet at the end of each month that’s just depressing.

For years, we’ve used a CSV generator to get the basic info we need for our monthend reports.  But it’s not perfect.  There was a lot of manual data entry and lookups- stuff that should have been automated long ago.

And it’s my job to do things like that- but the idea of spending a few hours on a project that would not enhance the product for our users at all seemed selfish.  I know some large companies have people, or even whole groups, dedicated to making their own tools easier to use for their own employees, but we just aren’t big enough.

So, last week I decided to look into automating the process a bit more.  And it wasn’t a cakewalk.  Getting our site counts, individual user counts, and related data was easy.  But getting new info – particularly the hit count for the homepage of just relevant sites was a bit of a challenge.

The hit count data is stored in a different table than the rest- so a whole new set of queries were created and installed in the original process.  The trickiest part was getting just the homepage- when the database was first designed, they simply added a string ‘[home]’ to the front of the homepage for a site, instead of using a nice boolean flag.  Easy enough- the SQL just grabs a substring of the first 6 characters and matches it against ‘[home]’.  But there’s also the possibility that an admin has updated the title of their homepage at some point in the past month.  So there might be 2 or 3 or even 10 rows that match that criteria.  Now we have to create a loop over them and add up this count for each using a temporary ‘total’ variable.  Though now that I typed that, I think we might be able to use SUM(count) for the query (assuming we’ve already snagged the proper rows only)- I wonder which one is faster?

Either way, the project is finished and now the initial report listings can be generated on a region by region basis with one click.  There’s still some work that has to be manual (checking on trouble listings, adding comments for losses/wins/etc), but it should make for a much quicker report process.

Non breaking spaces are breaking my balls

We’re still working on the search function.  The little preview text section seems to be coming through properly from the back end now, but I kept getting weird ‘character not found’ placeholders in the text.

You know- the graphic that shows when the browser doesn’t recognize the character.  It’s a ? inside a little black diamond.  In this case, it was the ‘ ’   – a non breaking space character.

I’d written a few regex checks in the php code to strip out certain things from the search results.  Anything inside style or script tags and any html code. Ok,fine – in truth, I hacked together these regexs in a terrible fashion.  There were angle brackets and backslash body parts all over my monitor.

Then, I threw the code sanitized results through a html_entity_decode call to make sure everything looks right.  But those stupid ? diamonds were still there.

Naturally, I thought something was wrong with my regex checks.  I wasted too long trying to rewrite them before I thought to check into the default html_entity_decode function.  Turns out, it doesn’t decode the   character.  Also turns out, I should have read the php official documentation a little more closely, as it states:

the ‘ ‘ entity is not ASCII code 32 (which is stripped by trim()) but ASCII code 160 (0xa0) in the default ISO 8859-1 characterset.

I don’t really know what that means, except that it meant my function wasn’t working.  The nice part- the fix is easy, just do a str_replace to remove them, and make sure to do it in the right order!

Where were we?

That’s right- the search feature.

After a bit of a delay, we’re back to work.  The current issue: adding the placement of the search button.  We wanted 3 options- in the header, in the footer, and in the left menu.  This was complicated by the fact that there is a layout option that doesn’t include the left menu (the horizontal menu).  In the end, we decided that if the user has the horizontal menu enabled, and they try to select the left menu placement for the search button, it will simply default back to the header.  Should eliminate any confusion (probably not).

The database tables are updated, the php conditionals are in place- the location of the search button is now customizable by the user.  But it still doesn’t really work very well.  Currently, the function just uses a simple sql ‘Like’ clause with the search keyword(s) inserted in dynamically.  But it returns the whole text area where the keyword is found.  We’re trying to find a way to limit it to a certain number of words- should be a simple matter of getting x number of characters before and after the keyword, and then finding the nearest space to cut it off at a word.  Actually, as I typed that, I realized there is a better way.  First, check to see if the keyword appears in the first sentence- if so, return that sentence.  If not, grab back to the previous punctuation and forward to the next punctuation (or end of the area) and display that.

Time to try it out!

On another note- I’ve started experimenting with Django.  It seems really cool- the model to database table relationship seems quite intuitive, and the templating system has been fairly easy to use so far.  The difficult area (for me) has been the url mapping.  It looks like once it’s set up, it is a great feature, but figuring out the regex based system is not going well for me.

Searching

I find that I get sidetracked a bit too easily.

I should be working on an upgrade of the recordkeeping feature.  As stated before, we’re looking to add a lot more functionality, as well as a nicer user experience, and it’s all likely going to be pretty difficult with the existing code base.

But I thought we might start with a simple facelift on the search feature.  The current interface is clunky and confusing- opening results in an ugly new window on top of the site, fullscreen.  Clicking any result in the listing will reload the page behind the search screen- so in effect, it looks like it’s doing nothing at all.

So we started sprucing it up a bit- open in a small modal, ajax requests for the results instead of leaving the page, term highlighting, and more.  But the results were off.  Searches were returning pages that didn’t have the term- why?  Turns out the search function was also searching any embedded html code.  Search for ‘document’, and you get a listing of every page with a jquery function embedded.  Search for ‘style’ and get every page with custom css inlined (bad practice, I know, but having a wysiwyg toolbar is a necessity).

How did the original developer get around this?  Simple- they didn’t.  The search results just returned a list of pages- no preview text.  Users would see the list and click on any link to a page, but there was no guarantee that their keyword would actually be on that page (it could just be a term in the source code somewhere).

So that’s been the bulk of the work so far- sanitizing the search results from the db to remove any code.  That way, a user only sees relevant results.  I naively searched for a way to only get non-code text from the db using some SQL magic, but (as anyone with a tiny bit of knowledge should know), that doesn’t seem to be possible.  So we’re reduced to getting all the results, then using some PHP regex checks to remove stuff that looks like code (anything between <script> or <style> tags, for starters).  The final loop over the results in the PHP file checks if the row includes the search keyword after the checks for code- if not, it reduces the count by 1 in order to keep the ‘total results’ number correct.

Is this the best or most efficient way to do it?  Probably not, but it works for now.  And we still have a lot of work to do before getting back on the recordkeeping track!

Platinum Album

Responsive design is kind of hard.

Tools like Bootstrap are awesome.  They make a developer’s life much easier, because it takes some of the design element questions away.  But there’s a lot of css in those Bootstrap files.  And trying to customize it can be quite tricky.

For example: we’re currently trying to upgrade our photo album feature.  The current setup is straight out of 2003.  A grid (created using tables) of tiny images- click one for a round trip to the server to get the large image, which may be 4000X3000 pixels, if the site admin didn’t feel like resizing.  Want the next or previous image?  Another trip to the server.  On the first image or last image?  No looping- sorry.

So, we’re taking advantage of Bootstrap’s awesome modal element.  It’s so smooth with just a few tweaks (found that the header/body traditional layout wasn’t going to work with the way our images displayed- too much vertical space lost, so we moved the ‘header’ to a right column instead.  Think Facebook’s image viewer layout).  After we added our own ajax setup to get the images, a little looping logic to allow users to cycle through, and a bit of responsive design (and extra ‘prettying’) of the default grid of images, we were just about good to go.

Then I checked my phone.

One of the main goals with all these upgrades is to bring everything mobile friendly.  A dedicated app may be in the future, but it seems like one direction things are going is into making an entire site work on mobile almost as smoothly as a native app does.  So, while upgrading the album feature, we are working in some mobile aspects.  The grid resize was fairly easy, but there are some stray styles on the modal that just won’t cooperate.  Mainly: when resized to a small device, our image automatically goes left-aligned (it’s vertical and horizontal centered in large view).  The modal also expands past 100% of the viewport on a phone, which has to be fixed.  I’m sure there’s some style in the bootstrap.css file that I need to overwrite, but it’s playing hard to get.

The other hurdle is touch events.  I thought jQuery touch would work- and it partly worked great, but it had some bad side effects when interacting with our system.  For some reason, it wanted to reload each page with an ajax fade in effect.  However, on our php-driven system, important info is often transported in the url string (like the id of the album we’re viewing).  When we enabled jQuery touch, this broke- links would fade the screen out, then in, but not change the actual content.

So we found an alternative: TouchSwipe – it’s working pretty well so far.  It’s not as responsive (in the action sense, not in the sizing sense) as a native app would be, but a user can swipe left or right to get the prev/next image.  The way we structured our JS functions, it was also super simple to add (just included a .swipe function call, with returns for swipeLeft and swipeRight).

So we’re getting there.  Not quite up to standard yet, but with this release, we’ll be another huge step forward.

:First!

The DOM Tree Of Life!
The DOM Tree Of Life!

One problem (of many) I ran into while adding validation checks on our request form feature was limited to IE8.  The code worked great in other browsers- my function used find() to get the correct error notice wrapper to display and switched it on if a field was blank or otherwise invalid.  But not in IE8.

In that world, the function would show the error notice wrapper for every field.  I made a little progress by fixing my PHP code- it originally inserted an error wrapper on each form field- even ones that weren’t required.  I figured it didn’t matter, as they’re all ‘display: none’ by default, but my first step was to add a conditional to my PHP to ensure that error wrappers only appear attached to required form fields.

That didn’t solve the problem- though at least it reduced the amount of red I saw on my test form.

Still- anytime a form was submitted with any unfilled (but required) field, any required field below that in the form was marked as invalid (even if it had a valid value).  Only in IE8.

The answer was to use the :first selector (https://api.jquery.com/first-selector/).  It looks like IE9 and later (and all other browsers) will stop at the first instance when using find(), but 8 will not- it will get all matches.  However, when I added the :first selector to my function that processed each error, 8 fell into line with the rest of them.  It stopped popping error messages all the way down the tree.

Another example of the notion that development can go so well for the majority of a project, and one little problem can stall for way too long.  Should I have thought of the :first selector earlier?  Probably, but because the code worked everywhere else, it was a bit of a head-scratcher for me.

Hope this helps anyone with a similar issue.  Actually- what I really hope is that everyone drops support for IE8 and no one has to read about this!

Stop Bugging Me!

We’re nearing completion on the request form feature validation, but I need a bug zapper.

We were so proud of ourselves early in the week.  Everything was working.  The functions were a nice mix of plain JS and jQuery.  Feedback given to a user while they filled in the fields of a form was informative without being frustrating (finding that line proved to be tricky, but hopefully we nailed it).  We though it would be a matter of some final testing and a push live.

Then I checked it in IE8. The rest of the week has been a frustrating, but informative, experience. Lessons learned:

  1. If you’re going to use jQuery, use it.  It seemed like we should try to use plain JS when possible, but in reality, if you’re targeting as much backward compatibility as possible… Well- there was a reason jQuery was created.  It smooths out the rough edges between ie, Firefox, and Chrome.  It makes things just work!
  2. Always test in an old browser early.  We’d really love to just drop support for IE8, and likely will eventually, but not like this.  The checks almost work, even in IE8, but are just flawed enough to make the feature very hard to use in that browser.
  3. Modern, updated browsers will tolerate multiple elements with the same id tag.  Older ones will not.

Point 3 was a tough one.  Because the form is being generated via a php/mysql backend, we don’t know what it will look like.  Therefore, all the validation checks have to be done with just a general knowledge of what the final form will look like.  For example- the main issue was with checkboxes.  There may be 4 required checkbox groups, and 2 non-required.  We found a way around this by using a wrapper div around a checkbox group and adding a ‘required’ class tag via php if that group is marked as required in the database.  Works great, but only if the wrapper div has the same id as the checkbox group.  Again, not a problem for most, but IE8 did not appreciate this at all (and I know, it’s also bad practice).  I think we found a way around it (by removing a possibly unnecessary id tag on a sub element), but it was not easy to find why some really odd bugs were popping up in IE8.

If I were to have a lesson #4, it would be that jQuery is quite awesome.  I know it’s not the new hotness that some js frameworks are, but it’s super useful.  I had a couple document.getElementsByClassName calls- they all failed in IE8.  Switch it out with a jQuery dollar sign and we’re money!  JQ also provides an awesome iterator function: .each().  That really helped, as again- we didn’t know how many of each type of field we would be checking.  It also smooths out the ‘this’ keyword- items within the .each() function have exactly the ‘this’ you would expect them to- none of the global window ‘this’ garbage.

Hopefully next week will be a wrap up of a successful launch.  Then maybe some different topics!