Jumping in the Deep End

I was (kind of) wrong in an earlier post.

 

That pause was to allow everyone to recover from such a shocking confession.  Everyone ok?  Ok- now we can continue.

Earlier this year, I wrote a post about making sure you’ve made a proper copy of an array or object in Javascript if you need to keep the original around.  Javascript objects and arrays (and arrays are really just objects in JS anyway) are pass by reference- so just setting “let array2 = array1” won’t work- any edits to array2 will also affect array1 as they both reference the same array/object.

So, you create a ‘deep copy’ of the array- the easiest way in ES6 being using the spread operator.  Just call “let array2 = […array1]” and you’re good to go.  Working in an older environment?  Use “var array2 = array1.slice(0)”.  Up to that point, my earlier post was correct, but it didn’t dig deep enough.  My solution to shallow copies was too shallow.

In many web development situations, someone working on the front end is getting data from a database of some sort.  In a current project, this is coming from Django.  The data is JSON serialized, so we parse it and get an array of objects.  Using the deep copy method on the array, we can manipulate the data for display.  As an example: we get some blog entries- then we want to get a listing of year-month pairs with the number of posts that fall into those groupings.  I know we could get this info with another DB call, but it’s already there in our original http request, so some manipulation should work fine.  We take the ‘posted’ value, make sure we just have the year and month, and do a loop to count up the total instance of each.

But that data manipulation occurs on objects in an array.  I created a copy of the array, but each element in that copy still references the original object from the original array.  I call it reference copy hell- I thought I had a fresh copy to work on, and technically I did- the new array was not the same reference, but that fresh copy was packed full of references to the original objects.  When I edited the posted date for grouping and counting, I also altered the original and the date display on the page was altered.

To fix, I tried the functional programming route.  One of the beautiful aspects of JS is that you can pass around functions just like any other parameter.  Arrays even have built in functions for this, like map and forEach.  So, to ensure I didn’t alter the original array or any object within that array, I made:

function copyArrayOfObjects(objArr) {
  return [...objArr]
    .map(obj => {
      return Object.assign({}, obj);
    });
}

And it’s chock full of ES6 goodness.  This will be for an Angular project, so we’re running through a transpiler and that should take care of any compatibility issues.  This function just takes the original array, makes a copy of that, and maps objects within to brand new objects using Object.assign.  It probably needs some sanity checks- just to make sure it’s being passed an array of only objects.  That would be pretty simple, but not necessary just now.  The only time it will be used is getting an array of objects from the DB, but if it seems useful to extend later, I will definitely revisit.

As an aside- I’m not sure I found the best way to accomplish my original goal.  I wanted to take the original array of objects and extract:
1) an array of category titles and a count of posts that fall in each
and
2) an array of year-month pairs and the count of posts that fall in each.

I feel like there should be a way to use .reduce to do this, but never could figure it out (still trying!).  What I did was to create another helper function:

function aggregateArrayOfObjects(objArr, field) {
  let aggregated = [];
  let intermediateMap = {};
  
  objArr.forEach(item => {
    if(intermediateMap.hasOwnProperty(item[field])) {
      intermediateMap[item[field]] += 1;
    } else {
      intermediateMap[item[field]] = 1;
    }
  });
  
  for(let i in intermediateMap) {
    aggregated.push({name: i, total: intermediateMap[i]});
  }
  
  return aggregated;
}

It takes my array of objects and the field to aggregate.  First it generates an object that tallies up the count for each distinct instance of the ‘field’ string you pass in.  Then it returns an array with ‘name’ (for the title of the field instance) and ‘count’ (the total times it was found).  It feels inefficient to loop an array to make an object just to push all that back into an array, but it does work (and I’m on a bit of a schedule with this) so it will have to do for now.  With this function and my original copy helper (yay functional programming!), I can get what I want:

const archiveFirst = copyArrayOfObjects(test)
  .map(b => {
    b.posted = b.posted.slice(0, 7);
    return b;
  });
const archive = aggregateArrayOfObjects(archiveFirst, 'posted');

For the archive (year-month pairs) we just deep copy the original array of objects (in a variable “test” here), manipulate the posted field (it includes the day of the post, but this isn’t really useful in grouping as it will be different for every post), and pass that into the aggregate function with the title of the field to aggregate by.  I know this wouldn’t work for an array of objects if one of those objects also references an object (or array), but I also know my data doesn’t currently do that.  There is probably a solution using recursion to walk through the array copy, look for objects, then look for objects/arrays within that object, and convert to a deep copy, but at that point I’ll just pull in Lodash!

Want to play with any of these?  This link to repl.it (an awesome playground for testing code in just about any language) should work.

Taking the Slow Road

The JS30 course from master instructor Wes Bos is really cool.  It includes 30 small* projects that really only involve JS/HTML/CSS- no libraries, frameworks, etc.  I’ve been working as a developer for long enough that I know some of the info already, but even if I see the title of a video and say “Awesome- I know that one!”, I can still learn something useful.  That’s part of the beauty of coding- once you get past the basics of syntax and making things run, there are many different ways one program can be written (and with JS, even the syntax can be a point of departure: semicolons anyone?).  It’s always good to see how someone else does it- you’ll almost always learn something new.

Generally, I’ll check out the first part of the video to see what we’re building, stop it, and go try to implement myself.  Then, I’ll watch the rest after either: A) I finish my version, or B) I get to a point where I’m not sure where to go next.

Example: one of the videos covers creating a cool sliding panel full screen image viewer.  My original version worked (using simple colors instead of images as a background), but I did not know you could animate the flex property on an element.  That’s how Wes did it in the rest of the video, so when I went back to re-do the project, I tried that instead.  It made the code much cleaner- in my original version, the css width values for the panel, grow, and shrink classes would have to be updated if I wanted to have 4 panels instead of 3- and the transition never quite had the exact width property (if you switch hover fast enough, you see the background image).  Also, I know animating width is generally not a great idea for performance as it triggers the unholy trinity of layout, paint, and compositing in the browser.  A quick check of https://csstriggers.com/ tells me flex does the same, but I tried using a more performance-friendly transform (scale) and it just did not look right, so a little less than 60fps will have to do!

If there’s one takeaway from this post, it should be that there are many paths to a working program.  I definitely don’t always find the best way on the first try- but if I approach the problem with the mindset of getting it working, then iterating until it works really well, I usually get the best results.

If there’s another takeaway, it’s that Wes’ courses are awesome (I learned most of what I know about React from his great course on that topic).  Some are free, some are paid- all are well worth doing.

But the final takeaway (I know, that’s three now) involves that asterisk from the very beginning of this post.  The idea behind JS30 is to do one project per day for 30 days.  And if you stick to just the parameters of what Wes lays out, you can definitely do that.  But it’s so easy to get carried away.  Is a program ever really done?  Eventually, you get to a point where it works and works well and you ship it- but it seems like there are always improvements that can be made or new techniques to be learned.  At work, you usually have someone reminding you that there is a deadline for a project, but with side projects, it’s easy to give in to scope creep.

Sometimes it’s not too bad.  For my updated version of the sliding panels gallery project above, I did a little mobile version that simply switches the flex direction to column and transforms to horizontal.  I also added a slight ‘screen’ effect over each image to allow for the text to show a bit better on the images I chose (learned this can be accomplished using a linear gradient as the background image).

//creates a soft darkening on the bg image without affecting the text color as opacity would
background-image: linear-gradient(rgba(0,0,0,0.3),rgba(0,0,0,0.3)), url('./img/tree.jpg');

On that same project, I decided I wanted a user to be able to click/tap any panel to have it open and have all others close (even if one is already open)- more of an accordion effect.  That was just a matter of adding a simple JS function:

function removeClassFromArray(elementArray, className) {
    elementArray.forEach(function(ele) {
        ele.classList.remove(className);
    });
}

I added it to the click handler for each, before the line where I set the ‘open-active’ class to the panel that should be in focus. This ensures only one panel is open at a time.

So, nothing too crazy- still a one day project.  But then there was the Fetch API project.  I feel like I know Promises and working with async code pretty well, so initially this one was going quickly.  However, as I finished up, I realized there was so much left to do!  A user can get the city population, sure- but what then?  Well, adding the total result count and total population wasn’t too hard.  But the total population number looked off.  There were no commas- so I created a function to add them (likely point for future optimization):

function numWithCommas(num) {
    let strArray = String(num).split('');
    let withCommas = [];

    var count = 0;
    for(let i = strArray.length - 1; i >= 0; i -= 1) {
        if(count === 3) {
            withCommas.unshift(',');
            count = 0;
        }
        withCommas.unshift(strArray[i]);
        count += 1;
    }
    return withCommas.join('');
}

Cool, so now we have a running total of results and the total population from all the results.  That’s not super useful though.  Wikipedia has a pretty cool API, why not grab info from that if they click on any result.  That will provide a blurb and link to the full article.  And we should have some kind of notification or feedback when info loads or something goes wrong. And stick the input to the header if they scroll down a huge list of results.  And if we put a listener on the scroll event, we will need a debounce function to not harm performance (that one’s very interesting- might dig into it next week).

You see where this is headed.  The todo list on the Fetch project is still growing to this day (mobile, better animations, etc)- though I’ve moved on to other projects for now.  JS30 has become more like JS300, but I feel like I’m still abiding by the spirit of the exercise.  That is, daily exercise for your coding muscles.

Light My Firebase

Last month, I finally carved out some time to build my own personal website.  Way overdue- but anything I create instantly looks like a pile of crap, so I usually end up spending way too much time trying to tinker with the look.  What I really love is making things work.

I used Angular to continue practicing that framework- particularly with the new animations module.  It’s pretty fun and easy to use.

I decided to host on Google’s Firebase platform.  I always thought it was just a real-time database, but now they offer content hosting as well.  With their tools you can install locally, it makes deploying super easy (as in: type “firebase deploy” level easy).

But what about a backend?  I wanted a simple contact form, so I’d need a backend to process the email sending (submissions would also be stored in the real time database).  I love writing Node- but most of my experience so far has been with Express- I wanted to get into the basics.  http, fs, dns, etc (that last one isn’t a module, don’t try to require it into your project)- all very useful and all native to Node.  My first thought was to just make a simple node backend with just a couple endpoints: one for sending email (using the awesome nodemailer library) and one for returning all routes to the root (to serve my Angular project).

However, Firebase hosting doesn’t work like that.  It’s not so much a backend hosting as a serverless architecture.  Please note: I may be using that term incorrectly.  In this case, it just means that you don’t use http.createServer and listen for request events.  Nor do you import express and build an awesome rest API.  With Firebase hosting, you use functions.

And those functions really are basically just simple Javascript functions.  In your app’s main folder, create a functions folder.  You can have multiple files, but be sure to have an index.js file.  This is where you can require other modules- or if you’re just making a simple function (like mine to send email), insert your backend code there.

You will also have a package.json in your functions folder (I believe Firebase creates this for you when you initialize their Functions option)  It will have a few dependencies by default (firebase-admin so you can authenticate your app and firebase-functions so you can register your functions).  You can add other Node modules here as well- I added nodemailer.  These become available as ‘require’ able in your index.js.

I created my email sending function- most of it is configuring the sending options and the template for the email content, as well as some nodemailer related config.  The content isn’t important- it’s just titled sendFormHandler.  Once it’s done, I just register it with an event on my database (in this case, saving a new contact on form submission):

exports.sendFormViaEmail = functions.database.ref('/form_submissions/{pushId}').onWrite(event => {
    const newSubmission = event.data.val();
    
    return sendFormHandler(newSubmission.name, newSubmission.email, newSubmission.content);
});

exports is just a shortcut for module.exports. functions is a reference to require(‘functions’).  I grab a reference to the data (the event is the database write, the data.val() function gets the object submitted.  Then I can pass that info to my simple sendFormHandler and off we go!

I’ll still stick to writing backend code for more complex applications, but this serverless thing is pretty cool (if that’s what this qualifies as).  One mistake I made you might avoid with Firebase: double check the path you provide to functions.database.ref.  Initially I was passing just ‘/form_submissions’ – the endpoint I use for saving new submissions.  However, this gets all entries in the form_submissions object (I almost said ‘table’, but this is not sql!).  I spent more time than I want to admit logging the return value from that endpoint, trying to figure out how I could get a diff of the previous object and the updated object to single out the new one.  I found a property: event.data._delta that showed the new object, but the pushId is randomly generated, so I couldn’t figure a way to reliably select just that object.  However, adding ‘/{pushId}’ to the end of the url you pass takes care of all that for you.  It gets the most recent entry by that unique pushId- very useful! only new addition, but had trouble accessing properties (pushId is randomly generated).

I, Object

I might have mentioned this before, but I started programming with the more loosely typed languages.  Think PHP, Python, or the loosest of the loose, Javascript.

With PHP, you don’t really need to declare what type a variable will be- you can just declare:
$fortyTwo = “42”;

Python is similar, even simpler to declare:
fortyTwo = “42”

Now, if you need to change the type of a declared variable, you might have to re-cast it, but you still don’t have to provide a type annotation beforehand:
PHP: intval($fortyTwo); // $fortyTwo is now 42 – integer type
Python: int(fortyTwo) // fortyTwo is now 42 – integer type

Javascript muddies the waters a bit, as you can add a string directly to a number, but you’re probably not going to like the results:
const fortyTwo = “42”;
fortyTwo + 42; // returns “4242”

It’s safer to explicitly coerce the “type” yourself:
42 + Number(fortyTwo); // returns 84
or
42 + parseInt(fortyTwo); // also returns 84
or
42 + +fortyTwo; // a bit more confusing to read, but also returns 84

All basic stuff- but the point is that until I started working with C# earlier this year, I didn’t realize how complicated the very basic task of declaring a variable can be, and how deep into your program those complications can extend.

In a current project, we have a bit of code we use to send out an email notification.  A user submits a form on the frontend to an api endpoint, where a method on that endpoint processes the info and sends the email.  .NET has a base class for this: IEmailSender- you can extend it as needed (and the process of extending classes is a good topic for another time).

The email processing can be used to send a few different kinds of messages (password reset, sales contact form, support request form, etc).  We have different templates for each, with different variables you can replace with custom info (the user’s name, their support ticket id, a password reset token, etc).  So that template info is just passed into our SendEmailAsync function as an object.  First, we initialize the object:

support_email customer_vars = new support_email {
    first_name = loggedInUser.first_name,
    support_id = objModel.support_id,
    request_type_id = objModel.request_type_id,
    screen_id = objModel.screen_id,
    request_subject = objModel.request_subject,
    request_notes = objModel.request_notes
};

The loggedInUser object is the current user (retrieved via the Json Web Token provided on each request) and the objModel object is the info from the form they submitted.  Now we can pass the customer_vars object into our SendEmailAsync function so they can be used in the template.

But making SendEmailAsync reusable was tricky.  If a variable is going to be passed into a function in C#, you have to give it a type.  Giving it a generic ‘Object’ type won’t work- so usually you just create a simple class or interface, name your fields, and use that as the type.  You can also do useful tasks at this point, like declare required fields or even restrict what can be entered for a field:

public class support_email
{
    public string first_name { get; set; }
    public string support_id { get; set; }

    [Required]
    public string request_type_id { get; set; }

    [Required]
    public string screen_id { get; set; }

    [Required]
    public string request_subject { get; set; }

    [Required]
    public string request_notes { get; set; }
    
}

This works great, until you want to pass a different type of object to the SendEmailAsync method. For example, we want to use the SendEmailAsync method to also send mail regarding a password reset. That’s not going to have a “screen_id” field, but you can’t just pass the wrong object type- C# won’t allow it.

This was an eye opener after a few years in Javascript-land.  There I can just pass anything as an argument and be off and running.  Our current answer is to create a super basic base class:

public class general_email_template {}

Then, each subset of email template simply inherits from that class:

public class reset_email : general_email_template
{
    //properties specific to reset email here
}

public class support_email : general_email_template
{
    //properties specific to support email here
}

And so on. In the SendEmailAsync method, we use the type “general_email_template” for our template variables object argument type and it works great. Any type that extends the type the method takes is acceptable.

Another possible option that we might explore is having multiple methods with the same name (SendEmailAsync) but different parameters. That way, we could just declare the specific type on each different SendEmailAsync version and let the language decide which one to use based on what it was passed (see previous post about how C# handles methods with the same name but different argument parameters). One approach might be better than the other, but this one works for now!

As an aside, another major difference between a statically typed language like C# and Javascript when performing a basic task is looping over an object. Coming from Javascript, it seems much more complicated in C#:

foreach(var v in variables.GetType().GetProperties())
{
    string prop_name = v.Name.ToString();
    string prop_value = v.GetValue(variables, null).ToString();
}

Than in JS:

for(let i in variables) {
    let propName = i;
    let propValue = variables[i];
}

The type system means you can be much more confident about what you’re getting as each variable, but it can definitely be an adjustment coming from a more relaxed language!

SPA Day

Picture yourself at a spa.  You’re relaxing in a nice hot tub, letting the worries of the week soak away into the scalding water.  Your phone rings – a flare of stress and responsibility is fired off into the night.  Do you really want to get out and answer that call?

Now, I’ve never been to a spa, but I have been in a couple hot tubs in my day and the idea of popping out into the cold just to answer a call is not appealing.  But, metaphorically speaking, this is exactly what I’d made our Angular application do.  It never complained, but I still feel bad- it wasn’t very thoughtful of me!

Let me explain- the spa in this case is an acronym – Single Page Application.  You may be familiar with this term- it’s an app that basically lives on the client/browser.  The bulk of the app’s structure and logic/functionality is downloaded up front as a bundle of Javascript, then network calls (think AJAX) are used to get data as needed.  A different model from the traditional request-response website, where each page is fully fetched, data and all, from the server.  You trade off a possibly slower initial load time (though that’s becoming less of an issue with tools like the Angular CLI) for a lightning fast app after that first load because you rarely have to go back to the server for more info- it’s all there in your bundle.

So, the metaphor does kind of work.  I’m the application, the hot tub is your browser, and once it’s loaded, why make it leave?  Of course, sometimes it has to make an important call.  If anything is going to be saved to a database or read from a database you’ll have to make that network call.  But the more you can minimize these, the faster your app will be.

On many views of this SPA, we show a grid of data.  That’s a trip to the database- no way out of that one.  There’s also an “edit” link on each row of the grid so a user can update the details contained within.  Initially, we had these going to the database to get the details of that row.  But that’s the old way of thinking!  That trip isn’t necessary to get anything to be displayed on the screen- it’s just to get data that’s technically already available (in the array of objects that’s fetched to populate the main grid).  All we have to do is pass it along to the individual view.

For our Angular application, I initially thought this might be an option on the Router class.  You can pass params to a child route, but that doesn’t really work in this case.  The data needed by the edit screen (child route) can be quite a few properties long, and passing all that in the query string didn’t seem like a great idea.  But this (sharing data between two components) seems like exactly the kind of thing services were made to do!

In the grid component’s template, there is an “edit” button.  When clicked, it fires a method on the component’s class to navigate to the edit component route and originally just passed the id/key of that item.  Now it’s been updated to pass the key and the object itself- the info from that row in the grid.  This object is passed to a service that has some other utility functions on it, but the one we need in this case is super simple:

setCurrentEditObj(obj) {
    this.currentEditObj = obj;
}

The currentEditObj is a class variable that’s just used to hold the object we’re about to edit.  This simple method updates that object any time the “edit” button is clicked on a grid.

In the edit component’s class, we subscribe to the route’s params observable.  This fires off info when the route resolves and gives back any params passed on the route.  In this case, it’s usually just the id/key of whatever is being edited.  In the original version, that id is then used in a getItem method- which just does an http call to the database to get the relevant info.  We inserted a check in that observable:

ngOnInit() {
    this.subscription = this._route.params.subscribe((params: Params) => {
        const currentObj = this.setupService.currentEditObj;
        if(currentObj) {
            this.countryName = currentObj['country_name'];
            this.id = currentObj['country_key'];
            this.setFormFieldDefaults(currentObj);
        } else {
            this.id = params['id'];
            this.getItem(this.id);
        }
    });
}

We wait for the route to resolve and check if the service has a currentEditObj available.  If so, we set the class variables (countryName and id for this view, which just edits a list of countries), and populate the form (setFormFieldDefaults just loops through the object, checks to be sure the object’s property exists on the form, and sets the initial value).  No database trip- our app gets to stay in the warm hot tub that is your browser!

But there’s one check you should do.  You might have noticed the “else” branch in the code above.  The internet isn’t perfect- your connection might be dropped and a full page refresh might be required.  A user might accidentally refresh the screen as well.  If that happens, it can cause issues for a single page application, because the object we are editing is now stored in the code on your browser.  If that browser refreshes, that info is lost.

As an aside – this kind of relates to the notion of offline availability.  Some really cool new techniques are being explored in this area using service workers– these can sit on your browser and respond to requests as if they were a server when a server isn’t available, but will be much faster (as they don’t actually have to travel anywhere) and be available if you lose your connection.  Very cool stuff, but not widely supported at this point.

Instead, we just include a simple check.  If that currentObj doesn’t exist, fall back to the old method and use the id from the url (which will still be available, as it’s stored in the url string) to go to the database and get the necessary info (getItem just does that, then calls the setFormFieldDefaults method).  The only time this should be called is if there’s a refresh or dropped connection, but it works nicely- the data is still available, and the extra cost of a database call isn’t really felt, as the entire application needs to be re-fetched anyway.

Does it save a lot of time?  Probably not- none of the objects we are fetching are huge.  But it is a saved network call, which definitely conforms to the spirit of a single page application.

Should I have thought of this originally?  Probably- it really makes more sense within the spirit of a single page application.  Anywhere you make an HTTP call should be examined and evaluated.  Does this call need to be made?  Do I already have access to this info?  Am I making my app leave the relaxing waters of your browser to make that call?  Let the app relax!

Have You Ever Danced with the Format in the Pale Moonlight?

Last time on The DevBlog…

We were formatting some data for display, but it also has to be searchable and sortable by the expected parameters.  Everything was working, but now an additional task appears- add commas to large numbers.

An example is a good place to start.  The column in the grid is for Current Shares.  A person might have 1,021,400.00 shares of something (Beanie Babies, maybe?  Doesn’t matter at this point).  The number isn’t stored in the database as a string- that would make my formatting job easier, but would make calculations on shares a bit unpredictable to say the least.  So, we will get something like 1021400 from the database and format it for display.

Perfect use case for Angular’s currency pipe:

this.items.forEach(item => {
    item['currentShares'] = this.currencyPipe
        .transform(item['currentShares'], 'USD', false, '4.2-2')
        .slice(3);

Again- we are doing a good deal of manipulating back and forth, so just using the easy route of piping in the template won’t work.  The code above loops over each item in the array and converts the ‘currentShares’ value to display as currency.

One small note here: shares aren’t actually currency, so we don’t want the ‘$’ or ‘USD’ to display, but the currency pipe is still very useful for adding commas.  We can just .slice(3) at the end of the chain to remove the ‘USD’ (the first 3 letters) from the string the pipe returns.  I didn’t find a built in method with the pipe to have no currency type display (it can display letters or a symbol, depending on the 2nd parameter passed), but this works quite well.

So far so good.  Our output is now showing as 1,021,400.00 in the grid (the two decimals are for consistency in display).  All done- maybe I can go home early!

But what about the sort?  Again- we’ve converted our number to a string.  In my previous post, we solved this by checking if: item[‘currentShares’] == +item[‘currentShares’] – but this won’t work here.  The string version has commas now- the above check returns false.

So a check was made at the call site of the orderBy function.  Before our array is sent off to the main method (in a shared service) to do the actual ordering, we check if the param (the property we want to sort by) is ‘currentShares’.  If so, we set a flag that is initialized to false (sortingByCurrentShares) at the top of our method over to true (more on that below) and loop through the items, stripping commas:

if(param === 'currentShares') {
    sortingByCurrentShares = true;
    itemsToOrder.forEach(item => {
        item['currentShares'] = item['currentShares'].replace(/,/gi, '');
    });
}

A super simple regex (the only kind I understand or trust) removes commas and replaces with nothing.  Side note: if you’re as hopeless with regex as I am, test them here – it’s a super useful tool!

Now we’re good on the sort.  The string version we are passing will “equal” (using the == operator) the number version and a numerical sort will work.  But now we need to revert to the string for output back to the grid.  In comes the sortingByCurrentShares boolean flag we flipped earlier:

if(sortingByCurrentShares) {
    this.items.forEach(item => {
        item['currentShares'] = this.currencyPipe
            .transform(item['currentShares'], 'USD', false, '4.2-2')
            .slice(3);
        });
}

The same basic logic as our original conversion (meaning I can probably move this out to a shared method- more fun refactoring ahead) is applied to convert back to the proper display string.  The user gets to see commas in their numbers.  The sort method gets to order numerically.  I get to check off another item in my task list.  Everyone is happy!

The Formatting Dance is Your Chance to Write Some Bugs

I love Javascript- but formatting numbers can be tricky.

Take money for example.  The Angular tracking application we’re working on has a grid to display data- and some of the columns contain dollar totals.  So, in our ngFor loop, we can just output the number in most cases.  You could even tack a dollar sign on the front: ${{data.money}}.  Works great if the amount is 19.99, but there are no trailing zeros allowed in numbers in JS, so if the amount is 20.00, the last two zeros are trimmed, leaving you with 20.

Not a big deal, but not exactly what people expect to see either- making the grid easy to scan and reducing confusion is very important in this app.  Not a problem- Angular comes with pipes.  It even has a built in pipe for currency.  Just pass your data, a ‘pipe’ character, and some config options in your template and you’re good to go:

{{data.money | currency:’USD’:true:’1.2-2′}}

That would tell your template it’s US dollars, show the $ character, and display at least 1 number to the left of the decimal and at least 2 but no more than 2 numbers to the right (this will round your number so be careful!).

Great!  Now it’s displaying as one would expect.  But we also have a search function on the grid.  Type in the input box and see your grid live update- thanks Angular magic!  But with a pipe, the data being displayed doesn’t match the data being searched through.  What was a number (5) is now a string (‘5.00’).  Search for ‘5’ and it works great, but a user may see the rest of the string and try searching for ‘5.00’- which will not work in this case.  No results found and a confused user.

Luckily, the awesome folks at Angular gave us an option for that as well.  Pipes can be used in a component instead of in a template.  Just import the pipe(s) you want to use at the top of your component file, inject them in your controller, and you can pipe away right in your methods.  It’s not quite as easy as just putting the logic in the template, but is a great option in this case.  We created a formatDataForSearch method:

formatDataForSearch(data) {
    data.map(d => {
        //loop through and convert specific items that need it- others remain as-is
        d.rate = this.numberPipe.transform(d.rate, '1.4-4');
        d.originalCost = this.numberPipe.transform(d.originalCost, '1.2-2');
        d.costBasis = this.numberPipe.transform(d.costBasis, '1.2-2');
    });
}

Works great- the data is formatted in the array that will be passed to our search function and everyone’s happy.  A user can type anything they see in the grid and search will find it, but the actual info stored in the database is the correct number type.

But wait- there’s one more problem.  We also have a sort function on the grid.  Click any table header and the grid live-sorts using that data.  I’ve chronicled my journey from buggy, terrible sort to less buggy, mediocre sort in a few blog posts, but the actual function is at a point where it works pretty well.  However, a string sort does not do the same thing as a number sort.  Pass [100, 2] to a sorting function and you’ll get [2, 100] (assuming an ascending order sort).  Pass [‘100’, ‘2’] to a sorting function and you’ll get [‘100’, ‘2’].  Why?  According to MDN’s entry on Array.prototype.sort(), “The default sort order is according to string Unicode code points”.

This means two things for our use case
1) If you don’t pass any comparison function to .sort, you’re probably ok for strings, but will not get the correct order for numbers.  Be sure to pass at least a simple comparison (a > b) will do for simple numbers.
2) Even with #1 covered, we will need to convert the data back to a number.  Otherwise, 100 will come before 2 in our grid and chaos reigns!

So, one more edit to our orderby.service.ts (which houses our sorting logic) was in order (ha).  And it gave me one of the very few legitimate (I think) use cases for == I’ve had in a couple years of writing Javascript.  We added a check in our sorting method’s data formatting logic:

if(a[param] == +a[param]) {
    first = +a[param];
    second = +b[param];
}

The first and second variables are initialized a little further up in the code- this conditional comes in a block checking the type of the data being sorted.  a and b are the things being compared in each loop of the .sort function (objects- with the ‘param’ variable being the title text of the clicked column in the grid).  So, if a comparison of the data in the sort column and that same data converted to a number (the ‘+’ operator is just a quick way of converting a number to a string) is truthy, we convert both back to numbers and compare.  The original display remains a string, so we keep those padded zeros in $5.00, but the grid sorts properly.

I’ve always avoided the double equals check, as it doesn’t really do a true comparison.  It does some type coercion and can give you unexpected results (0 == false, for example).  It’s generally safer to use the strict comparison operator (===) but type coercion does seem to have it’s use case.

What started off as a simple edit request in my job queue (display this number data in this specific way) became a rabbit hole of conversions and back again. To be honest, it was pretty fun.  And the journey wasn’t quite over.  I was feeling pretty good after this victory (however small), when I got an update on this job: “The current shares column should be formatted with commas.  For example: 2,004,100.00.”

Next week- the saga continues!  I can use the currency pipe to convert and add the commas, but those commas are going to break my nice x == +x check in the sort function.  Spoiler: my solution is ugly, but it does work- stay tuned!

Still haven’t found what I’m searching for

Just a quick TIL post this week.  Don’t worry, I’ve got another rambler lined up for next week involving converting data for display then to another form for sorting, then back again- what a ride!

I thought the search function was complete.  It had a bit of complexity to format the content being searched correctly, but the ‘guts’ were fairly simple, using Javascript’s built in string.search method to identify a the submitted substring within a larger object’s group of properties.  This is for an Angular application, so throw it all in an observable and watch your grid/list filter as you type!

Side note: we are only searching through content that’s already been loaded from the server, so it’s nice and fast, but if we were hitting the server on search, there’s a great rxJs operator- debounceTime.  Pass it a number and it will wait that period (in milliseconds) after your event before actually firing the action.  Basically, chain it to your subscription before your .subscribe operator and you’re good to go!  Super simple client side rate limiting!

Back on topic: the search worked, except when I tested with a special character.  The dollar symbol (“$”) in this case.  The grid had a column where this symbol would regularly appear in the text, but each time I tried to search for “MM$vx”, I would get no results- even if that text appeared in my grid.

Turns out- string.search doesn’t appear to work with some (or maybe all?) special characters inside a string.  I did a quick Google search as to why, but didn’t come up with any specific reasons for this so I might be totally wrong, but I did test this out in an awesome little tool at repl.it – it’s great for quickly checking if code works like you think it should.


const test = 'U$11M'.toLowerCase();
const nope = test.search('u$11m');
const works = test.includes('u$11m');
console.log(nope); // -1 not found
console.log(works); // true
console.log(test, 'u$11m', test === 'u$11m'); //same string

Interactive version at https://repl.it/H5AJ/2

You’ll see the solution in that snippet too- switch to string.includes.  That method doesn’t have any issues with any symbol I tested so far, so that’s what we’re using now.  I’ll be honest- I didn’t even know string had an “includes” method- that’s why we were using search!

One thing to note: string.includes returns boolean, while string.search returns the index of your substring, so be sure to do your checks for the proper thing (particularly when using the all powerful === check).

Staying on Script

I like npm scripts.  I’m sure real programmers can memorize long strings of command line gibberish without even waking up the full 10% of our brain capacity movies tell me is all we have access to, but I mess those commands up all the time.

Don’t get me wrong- it’s fun to be able to cruise around a system on the command line.  You feel like you really know secret stuff.  I like that you don’t really even need a monitor to get real work done on a server- just a secure login method and some useful commands.  ssh in, check out where you are and where you can go (ls -la), change directories and do it again (cd var/www && ls -la).  See something you want to check out?  Cat is your friend (cat package.json).  See something you want to change?  Use Vim (vim package.json).  Though be sure to install it first (sudo apt-get vim -y) and be ready to look up the commands just to get around the page (note: I am not a Vim expert- I only use it when I have no other option).

But when it’s time to get to work, it’s really nice to just be able to type ‘npm start’ and have your dev environment spin up.  Particularly when you’re a less than stellar typist.

Npm scripts just go in your package.json file- in the appropriately named “scripts” object.  They can be extremely simple- for example, a project built with the angular cli automatically sets up a couple: “test”: “ng test”, “start”: “ng serve”.  These just make it easier to find those (admittedly already simple) commands.  A developer will know they can run “npm start”, but if they’re not familiar with the angular cli, they might not know about “ng serve” right away.  Npm scripts work well as aliases- unifying those commands under one umbrella- and one less thing for me to try to remember.

You can extend those simple commands as well.  I made a simple edit to my start command: “start”: “ng serve –open”.  This passes the ‘open’ flag to ng serve, automatically opening my default browser.  There are flags for many other options- check out the docs!

But where Npm scripts have really shined for me is in abstracting away any directory changes and environment settings that may have to be done in order to fire up other aspects of a project.  Sure- when you’re just working on the frontend, it’s easy to remember that you run “npm start” from the same directory your package.json resides.  But one project I’m working on uses .net core as a backend API.  It runs locally, so one option is to fire up the full Visual Studio program, but this feels big and slow compared to VS Code now.  Luckily, .net core now comes with a command line interface.  I can run the command “dotnet run”, and my backend dev server starts right up.

But there are complications.  This executable is not run from the same directory as package.json- it’s technically a completely different project in a completely different folder.  An environment variable also has to be set to “Development” in order for it to run properly.

But Npm scripts don’t care!  A script will just take your commands and run them.  So, my new script is:

“backend-start”: “cd ../path/to/dotnet/project/folder && set ASPNETCORE_ENVIRONMENT=Development && dotnet run”
The path really is longer- this is just the way a .net project is initialized.  The point is- instead of typing everything to the right of the colon, I just type everything to the left (from my package.json directory).  The Npm script changes directory, sets my env variable, and starts the process- pretty cool and useful!  I created another to publish the backend and can now forget those file paths and environment settings and other commands.  More brain space for me!

Javascript’s Days of Future Past

Working with ES6 features when writing Javascript is really great.  It’s a pretty big update to the language, and smooths out many of the rough edges.  My original hesitation around using the new features was browser support- though that’s really no longer an issue thanks to the major browsers now rapidly adopting the updates.  Plus, just about all my projects these days run through some kind of transpiler (Typescript/Webpack for Angular or Babel/Webpack for React), so even some of those old browsers can be targeted (and really, Webpack is not that hard!).

There is one project (still tinkering- don’t judge too harshly!) that necessitates writing old ES5 style syntax, and it’s not much fun.  We’re looking at bolting on a countdown timer feature to an older system (that is in the process of being massively upgraded, but that will take time).  It’s pretty simple, but gets pretty ugly around the string templating aspect.

Anyway, that’s the project that got me thinking about strings.  In the old days, if you needed to include a variable in your string, fragments went into quotes, then out of quotes with a ‘+’ operator to concatenate your variable, then back into quotes.  If something you were adding onto the string had to remain a number type, you had to be very careful not to add two of them together back to back (or instead of 1 + 1 = 2, you’d get ‘1’ + ‘1’ = ’11’).

The introduction of template strings with ES6 really made this smooth.  The syntax reminds me of writing PHP- start a string with a backtick, then include text as normal.  When you want to include a variable, or some evaluated Javascript inside that string, start with a dollar sign and wrap in curly braces: ${myAwesomeVariable}.  End the string with another backtick.

But there’s always a but.  No matter how well designed or tested or coded your software is, someone will find a way to break it, misuse it, or complain about it.  That even applies to a software language itself.  In this case, trying to use the latest and greatest (ES6) with the old and busted (SQL queries over my database schema) caused a breakdown (of my code and my mental state).

The case: I’m currently working to update on old PHP/MySQL application to run on Node.  I’m using the awesome mysql node library to make calls to the database.  The database information can’t be changed- the application already has many users, and just dumping their data and starting over is not an option.

The mysql library lets me send those SQL queries to my database and gives me back the data I need (it even has a great ‘escape’ function to improve security on those requests).  But it doesn’t fix the difficulties with strings in SQL, and it doesn’t fix a 15 year old naming conflict in our schema.

When the application was created, a column in some tables was dedicated to the order of items (menu items, or pages, etc).  Unfortunately, this column was titled “order”.  Also unfortunately, “order” is a reserved keyword in SQL (used to sort your results).  In the old PHP code, the query string was just concatenated together over many lines.  It’s really a forest of single quotes, periods (the PHP string concat operator), double quotes, and backticks.  It’s hard to follow, so I decided to use ES6 string templating in my awesome new update.  But every query that included “order” blew up in my face.  And SQL errors are not exactly helpful.  Think: “You have an error in your sql syntax around ‘some random place in my query string here'”.

Turns out, if you have a reserved keyword as a column title, you need to escape it in a query with backticks.  You might be able to use single quotes, I’m not sure, but either way that won’t work inside an ES6 template string (at least, it didn’t work when I tested).

So, for requests to those specific tables, it’s back to queries like: “SELECT * FROM my_table WHERE id = ” + id + ” AND  ‘order’ = ” + order;  That’s a simple example- one query has to get the next available “order” value (the table wasn’t set up as auto increment), so first we have to get the max(order), then insert that variable into the “order” column.  Again, ES6 is awesome, but I’ve somehow found a way to have to go back to the old days!