Have You Ever Danced with the Format in the Pale Moonlight?

Last time on The DevBlog…

We were formatting some data for display, but it also has to be searchable and sortable by the expected parameters.  Everything was working, but now an additional task appears- add commas to large numbers.

An example is a good place to start.  The column in the grid is for Current Shares.  A person might have 1,021,400.00 shares of something (Beanie Babies, maybe?  Doesn’t matter at this point).  The number isn’t stored in the database as a string- that would make my formatting job easier, but would make calculations on shares a bit unpredictable to say the least.  So, we will get something like 1021400 from the database and format it for display.

Perfect use case for Angular’s currency pipe:

this.items.forEach(item => {
    item['currentShares'] = this.currencyPipe
        .transform(item['currentShares'], 'USD', false, '4.2-2')
        .slice(3);

Again- we are doing a good deal of manipulating back and forth, so just using the easy route of piping in the template won’t work.  The code above loops over each item in the array and converts the ‘currentShares’ value to display as currency.

One small note here: shares aren’t actually currency, so we don’t want the ‘$’ or ‘USD’ to display, but the currency pipe is still very useful for adding commas.  We can just .slice(3) at the end of the chain to remove the ‘USD’ (the first 3 letters) from the string the pipe returns.  I didn’t find a built in method with the pipe to have no currency type display (it can display letters or a symbol, depending on the 2nd parameter passed), but this works quite well.

So far so good.  Our output is now showing as 1,021,400.00 in the grid (the two decimals are for consistency in display).  All done- maybe I can go home early!

But what about the sort?  Again- we’ve converted our number to a string.  In my previous post, we solved this by checking if: item[‘currentShares’] == +item[‘currentShares’] – but this won’t work here.  The string version has commas now- the above check returns false.

So a check was made at the call site of the orderBy function.  Before our array is sent off to the main method (in a shared service) to do the actual ordering, we check if the param (the property we want to sort by) is ‘currentShares’.  If so, we set a flag that is initialized to false (sortingByCurrentShares) at the top of our method over to true (more on that below) and loop through the items, stripping commas:

if(param === 'currentShares') {
    sortingByCurrentShares = true;
    itemsToOrder.forEach(item => {
        item['currentShares'] = item['currentShares'].replace(/,/gi, '');
    });
}

A super simple regex (the only kind I understand or trust) removes commas and replaces with nothing.  Side note: if you’re as hopeless with regex as I am, test them here – it’s a super useful tool!

Now we’re good on the sort.  The string version we are passing will “equal” (using the == operator) the number version and a numerical sort will work.  But now we need to revert to the string for output back to the grid.  In comes the sortingByCurrentShares boolean flag we flipped earlier:

if(sortingByCurrentShares) {
    this.items.forEach(item => {
        item['currentShares'] = this.currencyPipe
            .transform(item['currentShares'], 'USD', false, '4.2-2')
            .slice(3);
        });
}

The same basic logic as our original conversion (meaning I can probably move this out to a shared method- more fun refactoring ahead) is applied to convert back to the proper display string.  The user gets to see commas in their numbers.  The sort method gets to order numerically.  I get to check off another item in my task list.  Everyone is happy!

The Formatting Dance is Your Chance to Write Some Bugs

I love Javascript- but formatting numbers can be tricky.

Take money for example.  The Angular tracking application we’re working on has a grid to display data- and some of the columns contain dollar totals.  So, in our ngFor loop, we can just output the number in most cases.  You could even tack a dollar sign on the front: ${{data.money}}.  Works great if the amount is 19.99, but there are no trailing zeros allowed in numbers in JS, so if the amount is 20.00, the last two zeros are trimmed, leaving you with 20.

Not a big deal, but not exactly what people expect to see either- making the grid easy to scan and reducing confusion is very important in this app.  Not a problem- Angular comes with pipes.  It even has a built in pipe for currency.  Just pass your data, a ‘pipe’ character, and some config options in your template and you’re good to go:

{{data.money | currency:’USD’:true:’1.2-2′}}

That would tell your template it’s US dollars, show the $ character, and display at least 1 number to the left of the decimal and at least 2 but no more than 2 numbers to the right (this will round your number so be careful!).

Great!  Now it’s displaying as one would expect.  But we also have a search function on the grid.  Type in the input box and see your grid live update- thanks Angular magic!  But with a pipe, the data being displayed doesn’t match the data being searched through.  What was a number (5) is now a string (‘5.00’).  Search for ‘5’ and it works great, but a user may see the rest of the string and try searching for ‘5.00’- which will not work in this case.  No results found and a confused user.

Luckily, the awesome folks at Angular gave us an option for that as well.  Pipes can be used in a component instead of in a template.  Just import the pipe(s) you want to use at the top of your component file, inject them in your controller, and you can pipe away right in your methods.  It’s not quite as easy as just putting the logic in the template, but is a great option in this case.  We created a formatDataForSearch method:

formatDataForSearch(data) {
    data.map(d => {
        //loop through and convert specific items that need it- others remain as-is
        d.rate = this.numberPipe.transform(d.rate, '1.4-4');
        d.originalCost = this.numberPipe.transform(d.originalCost, '1.2-2');
        d.costBasis = this.numberPipe.transform(d.costBasis, '1.2-2');
    });
}

Works great- the data is formatted in the array that will be passed to our search function and everyone’s happy.  A user can type anything they see in the grid and search will find it, but the actual info stored in the database is the correct number type.

But wait- there’s one more problem.  We also have a sort function on the grid.  Click any table header and the grid live-sorts using that data.  I’ve chronicled my journey from buggy, terrible sort to less buggy, mediocre sort in a few blog posts, but the actual function is at a point where it works pretty well.  However, a string sort does not do the same thing as a number sort.  Pass [100, 2] to a sorting function and you’ll get [2, 100] (assuming an ascending order sort).  Pass [‘100’, ‘2’] to a sorting function and you’ll get [‘100’, ‘2’].  Why?  According to MDN’s entry on Array.prototype.sort(), “The default sort order is according to string Unicode code points”.

This means two things for our use case
1) If you don’t pass any comparison function to .sort, you’re probably ok for strings, but will not get the correct order for numbers.  Be sure to pass at least a simple comparison (a > b) will do for simple numbers.
2) Even with #1 covered, we will need to convert the data back to a number.  Otherwise, 100 will come before 2 in our grid and chaos reigns!

So, one more edit to our orderby.service.ts (which houses our sorting logic) was in order (ha).  And it gave me one of the very few legitimate (I think) use cases for == I’ve had in a couple years of writing Javascript.  We added a check in our sorting method’s data formatting logic:

if(a[param] == +a[param]) {
    first = +a[param];
    second = +b[param];
}

The first and second variables are initialized a little further up in the code- this conditional comes in a block checking the type of the data being sorted.  a and b are the things being compared in each loop of the .sort function (objects- with the ‘param’ variable being the title text of the clicked column in the grid).  So, if a comparison of the data in the sort column and that same data converted to a number (the ‘+’ operator is just a quick way of converting a number to a string) is truthy, we convert both back to numbers and compare.  The original display remains a string, so we keep those padded zeros in $5.00, but the grid sorts properly.

I’ve always avoided the double equals check, as it doesn’t really do a true comparison.  It does some type coercion and can give you unexpected results (0 == false, for example).  It’s generally safer to use the strict comparison operator (===) but type coercion does seem to have it’s use case.

What started off as a simple edit request in my job queue (display this number data in this specific way) became a rabbit hole of conversions and back again. To be honest, it was pretty fun.  And the journey wasn’t quite over.  I was feeling pretty good after this victory (however small), when I got an update on this job: “The current shares column should be formatted with commas.  For example: 2,004,100.00.”

Next week- the saga continues!  I can use the currency pipe to convert and add the commas, but those commas are going to break my nice x == +x check in the sort function.  Spoiler: my solution is ugly, but it does work- stay tuned!

The Refactor Factor 2: The Rerefactoring

A while ago, I wrote part one of this saga.  I can’t remember what it was about, and am too lazy to look it up, but the theme was refactoring.  I went back to some working code and made it (arguably) better.  At least slightly less crappy.  Today, I did it again.

The aspect of our application under the knife was the sorting function.  Most of our views are just table type grids of data.  These grids have headings (name, address, admin status, etc).  Click the heading and the grid sorts according to that criteria.

I’d created the sorting logic- generally pretty easy, I mean, Javascript has a .sort method built into Array.prototype- but there were complications.  The function would not know what type of data it would be sorting, so we will have to have branches for string and date and others.  Nor would it know the parameter(s) it would be sorting by.  It would have to toggle ascending vs descending sort (click the header once for asc, again for desc).  It would have to be pretty flexible with its inputs.

And- to be added currently, hence the need for the refactoring- the ability for a second sort criteria.  We found that with only one sort criteria (for example: name), the function worked great- as long as everything was unique.  However, what about a column like ‘admin status’.  That’s just a boolean field- only two possible values.  If a user clicked to sort by ‘admin status’, they’d get one listing the first time, but then might get a totally different listing on subsequent clicks.

To make a more stable sort order, we added a defaultOrder variable on each grid component’s class.  That defaultOrder would serve as the initial sort (when the view first loads), and be passed to the sorting service as the second criteria.  So, if a user sorts by admin status, the secondary sorting is done by the defaultOrder (example: name).  The output is consistent and less confusing.

Example time- and please know that I know this is still a bit of a mess.  As I refactored, I started breaking some of the logic out into helper functions, but found that this was creating more work and complexity than simply taking care of it within the .sort call.  I may take another look, but sometimes it just makes sense to write the code in a ‘top down’ type layout- though I know this isn’t often the case.

Version 1: A no good dirty mess (that still kind of works)

orderByParamAscending(param, group) {
    return group.sort((a, b): any => {
        if (typeof a[param] === 'number') {
        //numbers- simple subtraction sort
            return a[param] - b[param];
        } else if(param.indexOf('date') !== -1 || param.indexOf('time') !== -1) {
            //date from string- convert to date obj and compare
            const first = newDate(a[param]);
            const second = newDate(b[param]);
            return +first - +second;
        } else if (typeof a[param] === 'string') {
            //strings- check for empty
            let first = '', second = '';
            //convert to lowercase and compare number code
            if(a[param]) {
                first = a[param].toLowerCase();
            }
            if(b[param]) {
                second = b[param].toLowerCase();
            }
            if(first < second) {
                return -1;
            } else if(second < first) {
                return 1;
            }
            return 0;
        } else {
            //other (usually bool)
            if(a[param] < b[param]) {
                return -1;
            } else if(b[param] < a[param]) {
                return 1;
            }
            return 0;
        }
    });
}

I know- not pretty.  We’re using Typescript, which is awesome, but for a while I couldn’t get this to work.  TS would throw an error about a function (.sort in this case) not having a unified return value.  And that’s because it’s a bad function.  But you can shut Typescript up by providing the ‘any’ return type for a function- just know that this is an almost surefire indicator of bad code.

So- how could I make it better?  Well- instead of doing all those checks on each parameter, I could create a couple variables that would hold the value- then update them based on the type.  Also, .sort works well if you return 1, -1, or 0 depending on the result of your comparison function, so once the values are properly parsed, we can compare and return one of those 3:

Version 2: Slightly simpler, but still no secondary sort

orderByParamAscending(param, group) {
    return group.sort((a, b) => {
        let first = a[param], second = b[param];

        //check if this is a datetime param or string and parse so we can sort numerically

        if(param.indexOf('date') !== -1 || param.indexOf('time') !== -1) {
            first = +newDate(a[param]);
            second = +newDate(b[param]);
        } else if(typeof a[param] === 'string') {
            first = a[param].toLowerCase();
            second = b[param].toLowerCase();
        }

        //parsing done- time to compare

        if(first < second) {
            return -1;
        } else if(second < first) {
            return 1;
        }
        return 0;
    });
}

A bit easier to follow- and we can lose the ‘any’ return type- we are now always returning a number.  We handle all the formatting of the comparators (the ‘first’ and ‘second’ variables) before doing any comparing.

But the proposed additional functionality still isn’t there.  I initially thought it would just be a case of calling the function twice on the same array- the first time with the default sort param (‘name’ for example) to make sure the data is in the right order at the outset, then call it again with the new param (‘admin status’) to re-order, with the original as a fallack.  But this does not work- the first call has no carry-over to the second (which makes sense, but I had hopes for a simple solution!).

And, in the end, the solution wasn’t that complicated- it was just testing to make sure the sort returns the right order that was a hassle.  The only real changes were to the arguments passed and to the sorting logic at the bottom of the function- nesting the secondary sort into the initial in case of a ‘tie’.

Version 3: A little more complex than V2, but a little more functionality too!

orderByParamAscending(param, group, altParam) {
    return group.sort((a, b) => {
        let first = a[param], second = b[param];
        let altFirst = a[altParam], altSecond = b[altParam];
       

        //check if this is a datetime param or string and parse so we can sort numerically

        if(param.indexOf('date') !== -1 || param.indexOf('time') !== -1) {
            first = +newDate(a[param]);
            second = +newDate(b[param]);
        } else if(typeof a[param] === 'string') {
            first = a[param].toLowerCase();
            second = b[param].toLowerCase();
        }
        //do the same for the secondary sort parameter
        if(altParam.indexOf('date') !== -1 || altParam.indexOf('time') !== -1) {
            altFirst = +newDate(a[altParam]);
            altSecond = +newDate(b[altParam]);
        } else if(typeof a[altParam] === 'string') {
            altFirst = a[altParam].toLowerCase();
            altSecond = b[altParam].toLowerCase();
        }

        //parsing done- time to compare

        if(first < second) {
            return -1;
        } else if(second < first) {
            return 1;
        } else {
          if(altFirst < altSecond) {
               return -1;
          } else if(altSecond < altFirst) {
               return 1;
          }
          return 0;
    });
}

There are other aspects we added- a ‘sortDirection’ toggle, a check for null values passed to the sort to convert them to strings (the comparison was failing on null- but some values from the database were null), but this is the meat of the function.  And I’m sure there are still improvements to be made- programming seems a little more art than science sometimes.  It can be hard to tell when you’re ‘done’ with something.  I was ‘done’ on version 1, but version 3 is much better.  At some point, you just have to move on to the next task, or your product will never ship!