Fat Model in a Skinny View

Working with Django can be pretty fun.  I’m not quite as familiar with the patterns and recommended practices as with Javascript and it’s associated frameworks, but there is a cool mantra I learned recently: Fat Models – Skinny Views.

I’ve been working on creating an API for a product website/blog.  The front end is latest Angular, the back end uses Django to return JSON (in as close to a REST API as we can get).  Because my Python’s a bit rusty (our latest project used .NET as a backend), I was catching up with some Pluralsight videos.

Side note: Pluralsight is a great resource for learning more about many programming languages, frameworks, best practices, and so on.  I’m not an employee there, just a satisfied customer.  The series I was using to brush up on Django was called Django Fundamentals.

Anyway, the instructor mentioned a phrase that resonated with me: Fat models and skinny views.  Basically, keep your logic in methods on your models, and make your views responsible only for returning data as much as possible.  Until this point, we’d been haphazardly filling up our views with all the logic and it was a bit of a mess.

The fat models/skinny views pattern allows for more abstraction and reusability.  Example time!

One of the views is to simply get the blog data in JSON format for display.  If the request passes a ‘slug’ (the identifying string for an individual blog), we get just that blog entry.  Otherwise, we get all.  Nothing super complicated- but joins on a database table can be tricky in Django’s ORM (the layer on top of any SQL transaction).  The ORM is useful, but I just couldn’t get it to join our main blogs table with the category table and the authors table.  As a result, the frontend was just getting the author id and category id instead of the actual string name/title (because only the id was stored in the blogs table- in a Foreign Key relationship).

From a bit of digging into Django, I found that you can ‘prefetch’ that related info using the select_related method.  So our view started as:

def blogs(request, slug=""):
    if request.method == 'GET':
        if slug != "":
            entry = Blog.objects.select_related('category', 'author').get(slug=slug)
            return JsonResponse(json.dumps({"result": True, "msg": "Blogs Loaded", "data": data}), safe=False)
        else:
            blog_list = Blog.objects.filter(posted__lte=datetime.today()).order_by('-posted').select_related('category', 'author')
            return JsonResponse(json.dumps({"result": True, "msg": "Current Blogs Loaded", "data": data}), safe=False)

We check the request (only GET is allowed on this unprotected route- there’s a different one with JWT auth for updating/adding entries) and then check for existence of a slug.  If one is passed, get the individual entry with the related info prefetched.

But there was an issue.  On the frontend, we logged the JSON response, but no author or category info was included.  Just the data from the blog table.  I went back to the Django side and printed out my “entry” and “blog_list” variables- everything was there!  From a little more research, it seems that the above method works just fine if you’re using the default Django view template, but we weren’t- we just wanted JSON data passed to our Angular components for display.  It looks like when the entry/blog_list collection was converted to JSON, only the original object info was coming along for the ride.

But the related data (category and author) was available on the original entry/blog_list object.  So, we could just loop through and grab all the properties we need, add them to a simple Python dict, and return that.  With that solution, our view officially passed from slightly chubby into full on fat.  The loop logic was repeated in multiple places and multiple views- not good for future developers who might have to update this code (probably me!).

So, we made a slight update.  As an example, for the above code, just after we get the Blog.objects collection (either the individual blog or group of blogs), we call to a method on the model:

data = Blog.get_blog_and_join(entry)

That one’s for the individual entry. On the model, we created a new method to handle that logic:

def get_blog_and_join(entry):
    data = {"pk": entry.id, "title": entry.title, "slug": entry.slug, "author": entry.author.username, "body": entry.body, "posted": entry.posted.strftime("%Y-%m-%d")}
    if entry.category:
        data["category_name"] = entry.category.title
        data["category"] = entry.category.id
    else:
        data["category_name"] = "No Category"
    return data

This just loops through the entry passed (the Blog object returned by Django’s ORM with related author and category data) and assigns the values to properties in a new dict.  Note the author prop- it references entry.author.username.  This didn’t work in the original method of just returning a JSON encoded version of the Blog.Object returned by the ORM.  A little extra logic to check if a category exists (they’re optional in our setup), and now we have a simple object with all the info we need.  It can be returned as JSON and the front end team has their precious data!

Strippin’ Tags

A while back, I helped create a Django back end for an update of an existing website.  The website’s front end was AngularJs, and we didn’t really get how the two should integrate.  The result?  A bit of a mess.  Django template views inside Angular html template files.  We had to modify the bracket style used for Angular (as it conflicts with Django’s).  The folder structure was confusing, the code was confusing- it worked, but it wasn’t a great solution.

Now I understand what we should have done: use Django to create an API for the Angular front end.  Just return that sweet, sweet JSON to the app and have Angular do the templating.  We are in the process of this update (as well as migrating this one from AngularJS to Angular the Next Generation, and I ran into an interesting project.

The original site had a Django-driven blog.  This was not integrated into the main AngularJS app in any way- a user clicked the ‘news’ link and they were taken to a completely new page with a traditional Django blog setup.  It made creation and updating easy- we just used Django’s admin panel for new posts, and the default templating views to handle any sorting (by category, date, etc).

But it doesn’t really fit.  Having the blog as a module within the Angular (now to be v4.2.4) application makes more sense.  With our new API approach it will work, but it will take some extra work.

One aspect is the admin panel.  We won’t be using the built in Django admin panel- instead, we’ve created an Angular-driven admin panel.  Converting the data returned by Django’s ORM to JSON was a bit tricky at first, but it’s flowing smoothly at this point (might cover that in a future post, as it was a bit of a process).

Another aspect is searching through blog posts (on the user interface), and that’s where we come to the cool project of the week.  One of the benefits of doing this extra work is getting the hip “instant updates” feel of a single page app within our blog’s UI.  When someone types in the search bar, they see the blog list below filter immediately.

But I noticed some test posts were coming up in almost all search results.  They happened to be the test posts with images in them.  The reason?  We are using an HTML editor toolbar for the admin area.  When a site admin posts a new blog, they use this toolbar to format text, add links, or post images (not everyone posting will be a developer or have source code access).

The toolbar has a cool feature where it encodes any images uploaded to base-64 format.  According to Wikipedia,  “Base64 is a group of similar binary-to-text encoding schemes that represent binary data in an ASCII string format by translating it into a radix-64 representation.

I don’t really know what that means, but I do know it means I don’t have to set up an ‘images’ folder for my blog module and save a path to those images with a relation to the blog post in my database.  When it converts to base-64, it gives me a string of text that a modern browser can translate into an image.  That image src can be saved right along with the rest of the HTML body.

Very helpful – but back to the search feature.  It turns out that the search was iterating over everything stored in the blog’s body- including the html code.  I would have had to fix this anyway, but it became really apparent with the image issue.  Because the string of text representing an image was so long, it naturally contained matches to most strings I was searching (at least the simple ones people would start with).  It might not have mattered on a traditional request-response site- a person would type their whole search string then submit and avoid seeing the wrong results.  But in a live reload search, the problem was obvious.

My first thought was a terrible one: Why not store a plain text version of the blog body in the database?  That way, we just return it alongside the rest of the JSON data and use that for the search.  But that really doesn’t make sense- it’s just bloating our database tables with duplicate info and increasing the size of the JSON object returned on each request.

Then I remembered how cool Javascript can be- and that it comes with awesome built in array features like ‘map’.  So, why not return the data as we have been (with the body as HTML) and manipulate the body just within the search function?  We can search over that manipulated body, but display the original HTML.

Turns out that this works pretty well.  In our main blog component, we initialize a search array on a service:

this.blogViewService.searchArray = ['author', 'title', 'body_plain_text', 'category'];

This is a convention we’re using on a different project- the array contains the property names we want to search by (those property names appear within each object in the array we will be searching over). That searchArray is passed to our search service with some other info.  In this case, ‘body_plain_text’ doesn’t exist on the object- but we’re going to create it as we go.

For the search, we cheat a bit and use Angular’s built in form input subscription.  A form input field in Angular has an observable you can hook into to get the data as a user types (called valueChanges).  You can then subscribe and do any searching there.  All we need to do is make sure to transform each object a bit in the process:

this.blogViewService.searchSubscription = this.blogViewService.term.valueChanges
    .debounceTime(200)
    .subscribe(result => {
        let itemsToSearch = this.blogViewService.originalItems
            .map(item => {
                item['body_plain_text'] = String(item['body']).replace(/<[^>]+>/gm, ' ');
                return item;
            });
        //we pass to another service to do the actual filtering of originalItems here
    });

We start with assigning a reference to this subscription (searchSubscription) so we can unsubscribe on the component’s destruction (to avoid memory leaks).  Then we hook into the form input observable (valueChanges).  I put a slight delay on the process at this point with debounceTime(200)- when the news/blog section gets big enough that returning it all up front doesn’t make sense, we will have to hit the database in this search.  debounceTime(timeHereInMs) is a great rxJS built in that handles debouncing your calls.

Finally, we get to the actual change- the originalItems array is mapped, but none of the properties within each object are actually changed.  Instead, we append a ‘body_plain_text’ property to each object that uses a regex to strip HTML tags.  Originally, it replaced with nothing, but then we had words joining together (if they were on the other side of tags), so replacing with a single blank space preserves the integrity of the search.  We never change the original ‘body’ property- this is where our HTML lives and is used for the actual display.

I’m sure there will be edge cases where this might not work and we have to tweak the process, but it’s a good start.  I also don’t think this is technically how you’re supposed to use .map- it’s a functional programming staple, and I’m using it to append a property to an object then return it back to the array.  Definitely not functional programming!

Nonbreaking Nospace Space

Recently, we’ve switched gears on the new project.  Still using Angular 2 as the front end, but the backend had to change- moving away from Python/Django to C#/.NET.  However, there was one cool feature/tricky quirk in Django I wanted to share.

Google plays a huge role in building a new website.  We definitely didn’t consider their ranking and site score index nearly enough in the planning stages.  As a result, we had multiple post-launch ‘fixes’ to the new site- all related to getting our site as Google-friendly as possible.

One of those involved minification.  Sure- everyone knows that minifying your Javascript and CSS files is important.  It keeps the overall size of each http transfer low- meaning: faster load times.  Add concatenation to that process and you’re also reducing the overall number of http transfers- meaning: faster load times (at least, until http/2 becomes the standard- it looks like that will allow for many more open connections during the transfer process- meaning: many things that have become standard about site speed best practices will change).

But Google will also knock your page speed score down if you don’t minify your html.  I hadn’t heard of that one before.  The size differences really didn’t seem like they’d make much difference- a few bytes here and there- but it doesn’t matter to the Google check.  If your html isn’t minified, you will lose points.

In steps Django.  There’s a great middleware feature: htmlmin (pretty sure you can just install it with pip and add it to the middleware array on your settings.py file).  It will automatically minify all your html templates.  Very useful- but there is one ‘gotcha’.

It’s possible that while you were creating your masterpiece of a website, that some ‘nonbreaking nospace’ spaces were inserted.  How does this happen?  I’m still not quite sure- generally it’s because a file has been saved in utf-8 (generally a good thing) but with BOM encoding (not sure what that is or why it really matters).  However, if you do have one of these stray characters in the minified version, it will create a blank white space on your site (usually at the very top)- the height of a normal character.

The offending character code is &#65279 – but you won’t be able to find it if you search your .html files.  You can delete every space on your file as well- it won’t help.  It appears that the only (or at least easiest) way to fix this is to open your file in notepad++ (a free download), change the encoding to simple utf-8 (no BOM) and save.

This won’t make any visible change to your html code, but it will eliminate the issue and remove the blank space.  And your html is minified!  And Google is happy!  Now I just need to optimize that 6kb icon image down to 4.75kb and we’re in SEO heaven!

A Greater Datetime

Django’s built in blog module is very cool.  Go back a few months’ entries on this blog and you’ll see that’s where I learned Django (and how most people do, I would think).  Powerful, well-documented, easy to extend- just a great thing to have available.

One of the features most blogs have these days is the ‘publish later’ options.  You can pre-create a bunch of entries, then have them post on certain days.  So it looks like I’m at my desk, being productive, while in reality I’m sitting on the beach, drink in hand.

Publish later isn’t built in to the default Django blog module, but it is pretty straightforward to add.  The default module does have a ‘date’ field- which allows you to set the date attached to the blog.  Why not use that?  Just update the view for your blog display to filter by the date field (set the filter to only show blogs with a date < today:

Blog.objects.filter(posted__lte=datetime.today()).order_by('-posted')

This also makes use of cool aspects of Django’s ORM- just throw a double underscore and ‘lte’ on a field and you have a SQL condition for ‘less that or equal to’.

There you go.  When creating a new blog entry, just set the date field (with built in calendar picker) to the date you want it to publish.  The entry is saved, but won’t display on your front end until the date you selected.

Simple?  Yes.  Impressive?  Probably not, but it was quick to set up and it works, so that’s good enough for me!

Drop the Base

It’s an unfortunate fact that some users out there are still on old versions of Internet Explorer.  Despite Microsoft’s recent efforts to finally put them down and sweep any memory of their existence away, developers still have to ask that hard question:

“Do I really need to support IE7/8/9?”

I really want to say ‘no’.  If we all just say no, everyone will have to upgrade and we can drop all the ugly hacks, workarounds, and so on from our code.

But that perfect world isn’t quite here yet.  So I got to spend some time making our Angular/Django app work on IE7/8 (Angular is pretty good with 9- though there is one cool/crazy bug below).  My answer?  Set up a conditional comment to check for IE < 9.  If true, a Django route takes over, redirecting the user of an antiquated browser to our back up page- just a basic notice of what we do, with a simple contact form.

Modern browser users will never see this ugly piece of crap.  Is it the best answer?  I don’t know- but it does work.  And it was kind of fun re-living some of the old hacks (I started developing more into the IE9 phase, but legacy support, blah blah blah).  Our lead didn’t want to use Jquery (I really like jQuery, so I resisted, but there are good reasons to leave it out of an Angular project- coming in a future blog!), so I had to go look up the old syntax for getting elements from the DOM.  I remember that getElementById works even that far back, but always forget that the entire back half of innerHTML is capitalized.  And actually- knowing I only needed to support IE meant that it wasn’t that hard not using jQuery.  You don’t get all the cool animation type stuff that’s built in, but from a functional standpoint, it worked pretty well.

Except one little thing.  Back to the IE9 bug.

Because we decided to let IE9 users access the full app (Angular officially supports 9), I didn’t set up a redirect- IE9 users go to the homepage and can access the whole shebang.  However, some of our Angular routes just weren’t working.  The view section was blank.  Checked the devtools and saw I was getting a 404 not found from Django on my partials that should have returned the proper template.  Worked fine in FF/Chrome/new IE, so I couldn’t for the life of me figure out what was going on.

Turns out, IE9 ignores the html <base> tag.  It seems to have no effect if you just drop it in your code as normal, so the links within the app weren’t using the proper relative path- they were trying to look in the same ‘folder’ (which didn’t exist).

After some Googling (thanks Stack Overflow), I found that answer above.  And a cool trick- just set the base tag, then directly below, add a simple JS function to add the base tag again.  Started working perfectly after that.

 

fix is easy js iife- but how long would it have taken me to figure out if not for the internet!

Drowning in a Sea of Dead Links

So there was a problem.  Shocking, I know.

The new site development is humming along (mostly)- and might be ready for launch soon.  The problem is the old url links.  The old site was not created using any kind of framework, it was just simple static html pages.  The new site has multiple frameworks that need to play nice together.  That’s not a problem, except with search engine optimization.  Apparently (and I’m no SEO expert), changing all your url routing can negatively affect your search results.  Links will be found that go to old, dead pages- or worse, nowhere.  Frameworks like Django and Angular don’t use the ‘.html’ suffix generally, so this could mean having to create a bunch of server url redirects.

Which gave me an idea.  Is it a good one?  Not sure, but it did seem easier and more efficient than a whole bunch of server redirects.

It goes like this: What is the Django urls.py file but a kind of layer on top of your web server?  It takes an incoming url string, matches it against a regex, and passes it along to the view.  So, why can’t we use that to redirect our old urls to the new method?

The answer is: we can!  One simple Django url setup:

url(r'^(P<path>[a-zA-Z0-9_-]+).html$', old_url_redirects)

And a slightly more complicated view setup (condensed for brevity’s sake):

def old_url_redirects(request, path):
    if path in {'new-route-one', 'new-route-two'}:
        category = 'new-middleman-here'
    return HttpResponseRedirect('/newfolder/'+category+'/'+path)

And there you have it!  The view function is a bit ugly because we reorganized the structure a bit to add categories into the url string (the old method just had one massive folder with all the endpoints).  If you don’t need that, you don’t need the ‘if path in’ conditional, and can remove the category variable from the return statement.  But the cool part is the ability to make a simple check over multiple options with a Python set.  So if someone gets the old link (http://www.mainsite.com/url-here.html), it will automatically redirect to the new route (http://www.mainsite.com/correct-category/url-here).

Does this method suck?  Maybe- but it does work!  If I’m missing something, or you see a more efficient way, help me out in the comments…

Play nice

The saga of Django-Angular continues!

We finally made a bit of a breakthrough.  I realized that Angular routing can work with Django routing just fine- it just takes a little of massaging both egos.  You see, no one likes to be left out, least of all Django, and that’s where my thinking was falling flat.  I thought we could just have the Angular routes take over, but where would they go?  Django is handling the ‘response’ part of any request in the http chain, so yes- we could set up Angular routes to handle some views, but still had to create those (literally in the views.py file) within Django.

So we added some $routeProvider.when() calls.  Cool trick I also learned- the templateUrl option can receive a function.  So really, we only needed one- pass the function the route params, then grab the current param off that and return it.  This way, we don’t have to set up individual routes for each path- one will do the trick:

when('/:id', {
  templateURL: function(params) {
    return 'path/to/your/file'+params.id;
  }
}

This awesome snippet brought to you by this badass: http://stackoverflow.com/a/15817458

Note that I’m leaving off the ‘.html’ that might go at the end of the return statement- Django doesn’t like when you tell it what type of file to get, it does that all on its own!  It will still work on another framework (or no framework), just add the ‘.html’ to the end of the string.

The final step was to appease Django.  First in urls.py:

url(r'path-start-here/(?P[a-zA-Z0-9_-]+^$', viewnamehere)

And finally, in views.py:

def viewnamehere(request, endpath=None):
  if(path):
    render(return_to_response('yourfolder/'+endpath+'.html', locals())
  return render_to_response('yourfolder/yourdefaultfile.html', locals())

I’m sure there’s something off here- it works, but only with one folder (if I try to make a 2nd url and view for a different folder, Angular never seems to pick it up).  So there’s more work to do, for sure, but it’s very useful for a project that doesn’t require too much deep organization.  It was also neat to learn a bit more about Django views- I didn’t realize you could pass the optional parameter (endpath=None) and have two outcomes for the same view (though, in retrospect, it does seem quite simple!).