Jumping in the Deep End

I was (kind of) wrong in an earlier post.

 

That pause was to allow everyone to recover from such a shocking confession.  Everyone ok?  Ok- now we can continue.

Earlier this year, I wrote a post about making sure you’ve made a proper copy of an array or object in Javascript if you need to keep the original around.  Javascript objects and arrays (and arrays are really just objects in JS anyway) are pass by reference- so just setting “let array2 = array1” won’t work- any edits to array2 will also affect array1 as they both reference the same array/object.

So, you create a ‘deep copy’ of the array- the easiest way in ES6 being using the spread operator.  Just call “let array2 = […array1]” and you’re good to go.  Working in an older environment?  Use “var array2 = array1.slice(0)”.  Up to that point, my earlier post was correct, but it didn’t dig deep enough.  My solution to shallow copies was too shallow.

In many web development situations, someone working on the front end is getting data from a database of some sort.  In a current project, this is coming from Django.  The data is JSON serialized, so we parse it and get an array of objects.  Using the deep copy method on the array, we can manipulate the data for display.  As an example: we get some blog entries- then we want to get a listing of year-month pairs with the number of posts that fall into those groupings.  I know we could get this info with another DB call, but it’s already there in our original http request, so some manipulation should work fine.  We take the ‘posted’ value, make sure we just have the year and month, and do a loop to count up the total instance of each.

But that data manipulation occurs on objects in an array.  I created a copy of the array, but each element in that copy still references the original object from the original array.  I call it reference copy hell- I thought I had a fresh copy to work on, and technically I did- the new array was not the same reference, but that fresh copy was packed full of references to the original objects.  When I edited the posted date for grouping and counting, I also altered the original and the date display on the page was altered.

To fix, I tried the functional programming route.  One of the beautiful aspects of JS is that you can pass around functions just like any other parameter.  Arrays even have built in functions for this, like map and forEach.  So, to ensure I didn’t alter the original array or any object within that array, I made:

function copyArrayOfObjects(objArr) {
  return [...objArr]
    .map(obj => {
      return Object.assign({}, obj);
    });
}

And it’s chock full of ES6 goodness.  This will be for an Angular project, so we’re running through a transpiler and that should take care of any compatibility issues.  This function just takes the original array, makes a copy of that, and maps objects within to brand new objects using Object.assign.  It probably needs some sanity checks- just to make sure it’s being passed an array of only objects.  That would be pretty simple, but not necessary just now.  The only time it will be used is getting an array of objects from the DB, but if it seems useful to extend later, I will definitely revisit.

As an aside- I’m not sure I found the best way to accomplish my original goal.  I wanted to take the original array of objects and extract:
1) an array of category titles and a count of posts that fall in each
and
2) an array of year-month pairs and the count of posts that fall in each.

I feel like there should be a way to use .reduce to do this, but never could figure it out (still trying!).  What I did was to create another helper function:

function aggregateArrayOfObjects(objArr, field) {
  let aggregated = [];
  let intermediateMap = {};
  
  objArr.forEach(item => {
    if(intermediateMap.hasOwnProperty(item[field])) {
      intermediateMap[item[field]] += 1;
    } else {
      intermediateMap[item[field]] = 1;
    }
  });
  
  for(let i in intermediateMap) {
    aggregated.push({name: i, total: intermediateMap[i]});
  }
  
  return aggregated;
}

It takes my array of objects and the field to aggregate.  First it generates an object that tallies up the count for each distinct instance of the ‘field’ string you pass in.  Then it returns an array with ‘name’ (for the title of the field instance) and ‘count’ (the total times it was found).  It feels inefficient to loop an array to make an object just to push all that back into an array, but it does work (and I’m on a bit of a schedule with this) so it will have to do for now.  With this function and my original copy helper (yay functional programming!), I can get what I want:

const archiveFirst = copyArrayOfObjects(test)
  .map(b => {
    b.posted = b.posted.slice(0, 7);
    return b;
  });
const archive = aggregateArrayOfObjects(archiveFirst, 'posted');

For the archive (year-month pairs) we just deep copy the original array of objects (in a variable “test” here), manipulate the posted field (it includes the day of the post, but this isn’t really useful in grouping as it will be different for every post), and pass that into the aggregate function with the title of the field to aggregate by.  I know this wouldn’t work for an array of objects if one of those objects also references an object (or array), but I also know my data doesn’t currently do that.  There is probably a solution using recursion to walk through the array copy, look for objects, then look for objects/arrays within that object, and convert to a deep copy, but at that point I’ll just pull in Lodash!

Want to play with any of these?  This link to repl.it (an awesome playground for testing code in just about any language) should work.

Page 2, or 22, or… Whatever

So the pagination service/component works.  Drop in the selector into your template, be sure to import the component in your module and the service in your parent component, and subscribe to the observable from the pagination service that gives you the updated page numbers as you filter/sort data, and you’re good to go.

But as the log grows, that little section at the bottom listing the page links gets longer and longer.  Even with just local testing, my audit log had over 20 pages (when viewing 10 items per page).  So we want to only show 5 pages at a time, and update that as you scroll through (so when you get to page 5, the indicator starts showing more).

Luckily, because we set up the page range as an observable, all the updates we needed to do were in one file- the pagination service.  If we could find a way to take the full pagination array (if you remember from last week, this is just an array of numbers- representing the first item of each page) and cut it down to 5 (easy), then update the contents based on the max and min (a little more complex), then ensure the edge cases (page 1 and page end) don’t break anything (a little more complex), we’d be good to go!

So, setting up the master array is the same process- get the data from the server, and generate an array with the number of the first item on each page.  For a 6 page listing and 10 items on each page, it might end up as: [1, 11, 21, 31, 41, 51].

Then, we find the index of the page we’re currently on:

let paginationArrayMid = this.paginationArray.indexOf(start);

Where ‘start’ is passed in on click of any page indicator (as the first item to be displayed in that page).  Once we get that, we just index back 2 and forward 3 (to account for zero indexing) and we have a 5 item array to pass back in the observable as the entry array.  But first, we have to check if we are on a page that would be too low to index back 2, or too high to index forward 3 (i.e.: the first 2 pages or the last 2 pages).  I used a simple conditional check and assignment- though I’m betting there’s a prettier way:


if(paginationArrayMid - 2 < 0) { 
    paginationArrayMid = 2; 
} else if(paginationArrayMid + 3 > totalPages) {
    paginationArrayMid = totalPages - 3;
}
let paginationArrayMin = paginationArrayMid - 2;
let paginationArrayMax = paginationArrayMid + 3;

Finally, we can update the array we want to show based on where we are in the array with a simple slice- with a check built in to make sure there are at least 5 pages, otherwise, just show the full array:


if(totalPages > this.maxPagesInListing) {
    this.paginationArrayDisplay = this.paginationArray.slice(paginationArrayMin, paginationArrayMax);
} else {
    this.paginationArrayDisplay = this.paginationArray;
}

The ‘this.maxPagesInListing’ variable is just a constant set to whatever you want the displayed page range to be (in our case, it’s 5- but is easily configurable). Then just update the observable and you’re good to go!


this._entryRange.next([start - 1, start + (this.maxItemsOnPage - 1)]);

Starting on a Fresh Page

The grid view needs pagination.  No surprise there- when creating something like an audit log, you usually don’t want to just dump all the data onto one screen- that could be thousands of entries.

So we use a service to automatically group the entries into pages.  We started with a method that will create a blank array and initialize it with ‘page 1’.  Then we find the total pages the array will have using the data that’s to be displayed.  Finally, we loop through the length of total pages and push a new number for each page onto the original array.  The number in the array will be the index of the first item on that page (from the master array of all log entries we got from the server).  I know- doing it this way will be slow on first load, and we may look into grabbing the first bunch of entries, then lazy loading the rest, but for now, we just grab them all.

this.paginationArray = [1];
let totalPages = Math.ceil(this.totalEntries / this.maxItemsOnPage);
for(let i = 1; i < totalPages; i += 1) {
    this.paginationArray.push(i * this.maxItemsOnPage + 1);
}

this.maxItemsOnPage is a number we initially set to 10- it just tracks how many entries we want to show on each page.  It’s stored in a variable because the user can change it- they can show 10, 25, or 50 items per page- each time they change that option, this variable updates and so does our pagination list.

Initially, we used Math.round to get the totalPages variable, but it caused a bug where sometimes the last page was omitted from the listing.  Math.round does a traditional round on a number- so if we came up with totalPages as 4.7, it was fine- rounds to 5.  But if it was 4.3, we lost the last page when it rounded down to 4.  Using Math.ceil ensures that anything over 4 rounds up to 5, and we have the correct number of pages.

In our HTML template, we just set up an *ngFor angular loop and we’re good to go.  To be honest- there’s more logic in the template than I like- things got a little complicated with tracking which page was the current page and adding/removing classes conditionally, but we will cover that later.

Short entry this week, but stay tuned for the exciting conclusion next week- where we limit the number of pages that show in the navigation to 5- then automatically update that as a user scrolls through the pages, revealing more as they go.

The Fault in our Select Default

One thing I haven’t quite figured out in Angular 2 yet relates to the ‘select’ element on forms.  Creating one by *ngFor – ing through a dataset makes sense, but every time the ‘selected’ option fails to set, leaving me with a drop down that appears empty until clicked.

What I wanted was a generic default option- like you’d normally do by setting the ‘selected’ property on the element.  Something like ‘Select Item Below’- so people know there’s the option, but the first element in the actual select array isn’t shown yet.  But it just didn’t seem to want to work.  Even if I manually added an option element above the *ngFor loop through the other option elements (provided by a call to the Angular controller), and set the ‘selected’ attribute on that first element, it just appeared in the list- it was not selected by default.

I’m betting there’s an easy way to do this, but I didn’t find it.  Instead, this is what I did in a ‘setDropdownOptions’ method on my component:

this.optionsUser = this.audit
    .map(entry => entry['user_key'])
    .filter((name, index, arr) => {
        return index === arr.indexOf(name);
    });
this.optionsUser.unshift('Choose User Email');

this.optionsUser is just an empty array to begin with.  The first part loops over the audit array to get the user_key (an email address), then return just that into the optionsUser array (the extra .filter call weeds out any duplicates).  Then, we unshift (add to the front of the array) the generic text for the default menu item.

One more step in the component- the select element is part of a FormControl Angular element.  We set this specific control to a variable called ‘this.termUser’ when initializing the form.  That way, once the array of ‘optionsUser’ is populated, and we unshift the generic title on the front, we can set the initial value:

this.termUser.setValue('');

Finally, in the template, we just loop over the optionsUser array as normal to display the available options from the database, but on each iteration, check to see if the option === our default text (‘Choose User Email’ in this case).  If so, set the value for this select element to an empty string (making it the default selected, but also meaning that if it’s submitted, it won’t alter our returned array at all):

<select class="form-control" formControlName="user_key">
    <option *ngFor="let item of optionsUser" value="{{item === 'Choose User Email' ? '' : item}}">
    {{item}}
    </option>
</select>

Does it work?  Yes.  Is it pretty?  Maybe.  Is there a better way?  Probably.  We’re working on abstracting it a bit more to make it a reusable option (instead of hard coding the default text, pass it as variable)- more on that later!

Otherwise, what?

Let me preface this by admitting I know next to nothing about SEO.  Those companies that advertise ‘top of Google’ rankings seem like modern day snake oil salesmen to me.  I mean, does Google even know how Google ranks search results?  Seems like a dumb question, but think about it.  What better way to keep your process secret than if you yourself don’t even know the whole thing?  Maybe that little input box is just a tiny bit self aware, and makes the final call on the order.

But it’s an important consideration when working with the web- particularly when working on a site for a company that depends heavily on search-related sales inquiries (everyone these days?).  So we created a nice, shiny Angular 1.5 driven site for a company.  It works, it’s quick, it’s got nice fades and slides and transitions.  All was well.

Until Google started indexing.  Or, rather, didn’t start indexing.  Turns out, despite what they’ve stated, the crawler just wasn’t rendering our JS before indexing a page (most likely an issue with how we structured our app, but the results were the same).  The company was dropping in search results, title and description tags were confusing strings of curly braces and dot notation variables- total anarchy.

So we turned to Prerender.io – and it was a great decision.  This post isn’t about Prerender, but they really are great- and when we had questions about implementation, they were very quick to respond.  After a few stumbles, we got it working and were climbing back up the ladder.

Until our cached links started to explode.  400, 500, even more- there definitely aren’t that many links on the site, and some were just complete gibberish.  Random strings of text- probably part of some automated Google crawling, but all were being cached by Prerender, because all were returning a 200 status message.

One of the limitations of Prerender with Angular is that you can’t use an ‘otherwise’ route (it uses a specific tag in your url to serve cached pages to search bots, post js rendering).  But that also means if you used dynamic routing (using the templateUrl function option to get the proper endpoint path from the url, then passing it to your backend), everything is returning status 200 ‘ok’.  And all knowing, all seeing Google is indexing everything.

Prerender is so awesome that they include a meta tag that lets you return a 404 http status, but how would we add it?  There’s only one html page with a ‘head’ in the whole damn project.

Our answer: dynamically add the tag to the head if our checks for valid routes fail.  Using the ‘onRouteChange’ hook in Angular’s lifecycle, we could add the tag anytime we served up our custom ‘not found’ notice (which isn’t actually a 404 page).  The last piece to the puzzle was finding a way around the missing ‘otherwise’ route.  We were using the path on the url itself to get the right endpoint from out (Django) API, which seemed really clever at the time, but now how do we deal with paths that are outside the range of our checking?

Arrays to the rescue!  We decided to grab the location.path() when a page loads (as from a search engine link) and split it into an array.  We knew the max number of items in this array should be 5, and knew what the last term should always be.  From there, it was an easy check to make sure that array[4] === ‘ourKnownLastTermConstant’ and that array.length <= 5.  Anything else gets a nice ‘not found’ message and the one html file gets the Prerender 404 meta tag added (which is then removed as the first step when clicking a different link, and the check starts all over again.

It was really only necessary to do the check on a full refresh- no one will get to a ‘not found’ page by clicking links within the app itself- those are all pre-validated in our routing.  So it shouldn’t be too big a hit on performance.

As is becoming a recurring theme on this blog, I’m sure this isn’t what the folks at Angular intended their tool to be used for.  But it works!  Different strokes for different folks, I guess.

 

Arrays In Action

I know these form validation entries are getting dull.  But it’s a major project and has led to some interesting/frustrating issues.

For example-  we wanted to be able to automatically scroll a user directly to the spot on the form where they had the error.  The simple way is to use input.focus – works fine if there’s just one error.  But there were some drawbacks:
1) Can’t seem to ‘focus’ a checkbox or radio group.
2) If multiple errors, it could focus one field, then another, then another in quick succession- a very confusing user experience.
3) The focus option didn’t scroll the screen high enough for the user to see the field label.  Sure, it’s easy to scroll up a bit to see what they need to enter, but again, not good user flow.

The answer? Arrays in action!

We created a resultArray variable- on each check in the $(form).on(‘submit’) handler, if something failed, we pushed that jquery object to our resultArray.  Then, at the end of all the checks, we $(body, html).animate() to 50px above the entry at resultArray[0] (found by getting resultArray[0].offset().top – 50).

And it seems to work great!  The array holds all errors- we can then scroll directly to whatever one is first in that array.  The project is about a week away from being done- so probably only one or two more entries on this.  Next week- the process of inserting ‘live’ checks, so a user can get real time feedback on fields that initially gave an error, but may have been fixed.  It was an interesting balance of enough feedback to let them know that ‘this field is now good to go’ with ‘stop giving me an update every time I type a damn key’.  Hopefully we got it right.