What do you want to learn?
Skip to main content
Node Application Patterns
by Rob Conery
Design and development patterns for building applications in Node.js.
Start CourseBookmarkAdd to Channel
Table of contents
If you're a Microsoft developer, and you've been working with .NET for awhile using Visual Studio, this might seem a little odd. Working in the terminal or the command line is not something that Microsoft develops normally do. However, working in Node or Ruby or Python or anything that's not Microsoft, you do end up working in the terminal. So let's just quickly take a look at some of the commands and some of the things I'll be doing in the terminal, because it will be a big tool that we'll be using. So this is the demo site that I just created, and I showed you with WebStorm. If I want to see the files in here, I have a shortcut called l, but if I hit ls -l, then it's the same thing, so ls -l, and it shows who owns what. It shows the size of the files. It shows the file names, when they were last updated. It shows you just about everything. Now why is this important? Well, if you want to navigate around and take a look at things, you certainly can. For instance, if I wanted to run the tests without involving WebStorm, let's just say I'm really quickly wanting to see are these tests, certain tests, running or failing or whatnot, I can do that. So let's go into our module where we put some tests, and I forgot the module name, but here I can see what it is. It's cart, and I come on in here, and I take a look and see there's the test directory. I also see that there's a new file called product.js, and I can take a look at that and, well, there's the code. By using the cat command, it just throws the file contents right out into the terminal window. Alright, well, let's run our test, and I do that by executing Mocha itself. It's an executable. It'll go in. Take a look at any test directory or a test.js file that you happen to have, and it found our test directory and ran the files inside of it. And, unfortunately, it didn't find any tests, which is weird because I thought I wrote some. What's going on here? Well, let's find out. We'll go into the test directory, take a look. Do we have any specs? We do. We have two. Product_spec and cart_spec. Well, I can do the same thing. Why isn't this showing up? And I can see here I've got no specs. What about the cart. I can see here---whoops, I can't go into it, sorry about that, fat fingers. And there, okay, so now I'm recalling that I backed all these things out. And these are just two files that have no specs in them. Well, this is all making sense. I was able to do this really quickly, and I was able to execute the tests and see that I had none. But I'm going to guess that maybe you are thinking, now that's neat, you're on a Mac, and it's a whole lot easier for you than it is for me. I'm running on Windows, and we don't really have the same kind of terminal you do. And I can dig that. But at the same time, it's not true. You actually have a better one. PowerShell is amazing. If you haven't spent any time working with PowerShell, please consider this to be a little prod from me to you. It is amazingly capable. So let's do something fun. Here's a Windows 7 VM I have running on my Mac, and what I'm going to do is to take the demo folder I've been working in and just drop it right in and there is it. Good. So now I have a demo folder, and I want to open up this demo folder in PowerShell. So I can come right up here into the address bar and type PowerShell and kaboom. PowerShell is open and in this directory. Good. I've taken the time to make this look as close to my shell on my Mac as I can. I'm actually a little bit jealous. It's an amazing tool. And just really quickly, to show you what I did. Head on into Properties, and inside of here, take a moment and set the colors, the background colors, to something pleasing to you. I just copied what I had here on my Mac, and I replaced the RGB values right over there. Simple to do. In addition, I set Consolas as my font. Set it to 20. It's sometimes between 20 and 24, so I can see it a little bit better. Make sure to bold the fonts as well. Anyway, that makes it just a little bit more of a pleasing environment. And what we can do in here is check our version of Node. We can do all the same things with PowerShell as we can with regular Bash. In fact, a lot of the commands, like ls -l works almost exactly the same. The "-" arguments evidently don't work, but ls will work. And you can see LastWriteTime, Length Name. Little bit different than what we saw over in Bash, but, anyway, it does work. So, what I can do here is say NPM install, and it's going to go into my package JSON. As you can tell, I have Node installed on my project here. And I can do roughly the same things. I can go into our lib directory, and I believe it was cart, and it corrected my forward slash to a backslash. That's pretty groovy. And then take a look around. There's our test, and I should be able to just run this. So let's do NPM. I don't have it installed globally. Install mocha -g. So it'll take just a second. Here we go. And now if I run Mocha in the test directory, same exact thing. It runs the test, and then if I want to take a look at the test files, I can come in and say take a look at---let's actually go into our test directory, and I will do cat, and then product spec, and same exact thing, which is really, really handy. So, point being, as I'm cruising along here in the command line, crack open PowerShell. As a matter of fact, there is a beginning course on PowerShell introduction here at Pluralsight. It is outstanding. I watched it for a bit. I couldn't believe some of the things you can do. To taunt you with it a little bit, you'll notice that I'm working on the C: drive here. One of the things that you can do with PowerShell is to mount individual directories as drives including your registry, which I think is pretty cool.
Setting Up Your Project
Setting Up Grunt
It's easy to say something along the lines of think in terms of modules when discussing Node and how to organize your code. But what does it mean? Here's one way that I personally have found very helpful. Straight upfront, I try not to have a parent project. In other words, if you're developing a web app, try to avoid putting all your logic inside of a single lib directory that in turn is in the root of your web app. What I've typically done is to create a single directory for my project and then subdirectories for each module that goes into it. But what are these modules? How do you know what should go into a module? This is not a straightforward answer. The best possible advice I could give is to not over-think it from the start. That's easy to say, but what does it mean? Let's walk through this. This project is an eCommerce project that I am building for fun, eventually to open source it or something else. Right off the bat, I know that I need a website and that I'll need modules to handle all kinds of things, from shopping cart interactions to checking out, from order fulfillment to customer management. And that's a lot of stuff. My goal with slicing the project up into modules is not so that I can reuse it later on. It's so that development can become tightly focused, and future maintenance becomes really simple. I'm going to borrow from what I know of domain-driven design and think about this project in terms of business services, also known as bounded contexts. So, that's how I'll start, and know this as well. You can always extract things out even more later on. I'll start with a membership module, as we'll have the notion of members, as well as customers. I'll also add sales, accounting, and fulfillment. You certainly don't need to structure your folders in this way. However, being the sole developer on this one right now, I prefer simplicity. So I'm going to keep everything right in front of me. What's most important is that each module is separated from the other. This is what I want. Now what I can do is drag package.json, Grunt file, and Node modules into the membership directory. These files are now dedicated to my membership module. Before going any further, I'll rename my module to something a bit more specific. For this, I'll add froggyfrog-membership knowing that I can rename it later. I'll also add an author and description while I'm here. As a side note, I should also have a version in here, but I overlooked it at the time I recorded this. But why did I do all of this? Node developers have adopted the Unix mantra, Do one thing and do it well. I want to do that here. To develop a membership module and hopefully do it well. That means I'll need to have models and processes specific to a membership system and to expose that to calling code through some type of API in my index.js file. I also want to have tests specific to this module contained directly in the module itself. This approach is different from a larger, all-in-one approach like you might see in Rails or .NET where your tests are in one place for all of your modules or projects. This makes things difficult to manage as a project grows and expands.
Choosing a Test Framework
Let's set up testing for our module. There are a few test frameworks to use, Nodeunit, Vows, QUnit, Jasmine. Those are some of the more popular ones. But most people I know use Mocha. I'll install Mocha globally using the -g flag. This installs the Mocha test runner and executable file into my path. I'll also install the Should assertion library making sure to save it in my dev dependencies. For many development frameworks, installing the test framework and runtime is the easy part. Getting everything configured and running and working correctly, well, that's the hard part. With the Mocha and Node, we add a test directory and our test files into that directory. Now we get to write a test, which I'll just stub out here with a bunch of silliness. (Typing) To see the results of this test, I just execute the Mocha command in the root of my module, and that's it. Tests run quite fast, but they don't always stay this fast, however. As your project goes, yes, it will take awhile to run all these tests, but nowhere near as long as an un-tweaked Rails RSpec setup. But, that is not the Node way. We want our module to do one thing and do it well. This means our test suite can stay smaller, and there's no large environment to load up. Looking at the output here from Mocha, it's okay, but it's not the nicest I've seen. Let's change that output by adding a few options to Mocha. I can do this as flags in the Mocha call itself or I can create a file called mocha.opts in my test directory. Running Mocha again, much better.
Flexing Git and Github
Let's get rid of this sample test file and commit our module to Git. This is very, very important. If you're working on a larger system as I am, you don't want to add the entire parent directory to Git. Each module should have its very own repository so that we can use NPM to load it. NPM works directly with Git and Github believe it or not. And you'll see this in detail later on. I'm going to add a .gitignore file here, and this is something you don't want to forget, as well. You don't want to commit your Node modules to your source tree. Never do this. This adds a huge amount of bulk to your repository and makes it hard for people to work with your module. Your module will contain modules that contain even more modules, and this can go on for a very long time. That is a lot of files. Node developers are used to working from the package.json file letting NPM install the dependencies for them using NPM Install. Let's add our files to Git. Oops, I forgot to initialize there. And we'll do our first commit. I've also added a remote up at Github so I can show you why it's important to have a single repository per module. Using NPM, I'll install the Express Web Framework and then ask Express to create a new site right here in our projects' root, which I'll call froggyweb. Next, I'll crack open our Express apps package.json and tell it that I have a dependency at tekpub/froggymembership. NPM has the ability to search for your module in a lot of places. As we've seen, it will look for it on your local drive if it sees a file or directory reference, which must start with a '.' or a slash. If your reference isn't a local one, NPM will assume that you want a module listed in the NPM registry up at registry.npmjs.org. If it can't find your module there, it will parse your module into a Github repo. I can tell it's trying to do this by reading the output of the command. So NPM found my repo and is about to install it, but we have a problem, one that I mentioned previously. I don't have a version number in my repo, which is very bad. This is how NPM knows which version to work with and will stop installation if one isn't specified because then it wouldn't even work at all. Let's fix that by adding a version number to our package.json and pushing back to Github. Now let's run the installer again, and it works. Taking a look at our Node modules directory, there's a module called froggyfrog-membership.
Building a Registration Module
A Pattern Discussion
Defining a Service
Dealing With Data
Saving the User Record
Pyramid of Doom
Building an Authentication Module at Full Speed
Again, in Real Time
Debugging Evented Code
Integrating Our Module into a Web App
Creating an API With Index.js
We're almost at the end of this course. This is the last module. All we have left to do is integrate what we've built into an Express Web Application and make sure it all works. We'll create an API for our Membership module so that other applications can use it. We'll set up another Express Web Application and then install our Membership module as you've seen me do using a file reference. Then, we'll hook up Passport, an Express middleware package that controls access and handles user authentication and authorization. After that, we are done. So far, I've been working on the core bits of the module without worrying too much about how it will be used. Let's fix that by layering on an API using an index.js page. One reason I like working with events is that consumers of your API can subscribe to events and do what they like. Given that, I need those events to ripple up through my API, so I need to make the API event driven as well. I have a handy template for that in WebStorm, which I'll use, and I'll call my API, surprise, Membership. The one difference here is that I'm going to allow calling code to pass in the name of the DB we need to connect to. This will likely change. As I mentioned, I don't like passing strings directly, but for now, this will work, so I'll leave it. I'll require my Registration and Authentication and then create an Authenticate method. And this is a tough choice. I just said that it's encouraged that you pass in a single object for your argument data. But this is a login routine, and I think for many people, it's common and somewhat expected to have a signature just like this. In just a minute when I plug this module into a web application, you'll see more of why I made my API this way. I'll connect to the DB and pass a live DB object to my Authentication module and then emit the events as they come back. (Typing) I was hoping to find a more elegant way to do this, but from what I can tell, this is the best way to ripple events up. Finally, I'll call Authenticate and pass along the callback. A quick note, you'll notice that I wired the events before I called Authenticate. You have to do it this way or you won't be subscribed to those events when they go off. It might seem obvious now that I've explained it, but it's not very intuitive when you're writing the code at development time, trust me. Let's do the same thing for Registration. And it's just occurred to me that I am not emitting anything when Registration is successful, so I'll fix that here as well. (Typing) Finally, I'll need a way to look up a user based on a cookie token, so I'll add that to the API too. I did things a bit backwards here. I don't like that. I don't have tests, so I'll -- those right now. (Typing) And there we go. Just passing. Now let's see how we can actually use this thing.
Plugging into a Web App
We are ready to work with our module in an external app, so I've changed the name of it to Membership and upped the version to 0.0.3. Whenever you make changes, you have to be sure you change this version. Otherwise NPM will ignore it when users go to update it. I'll install an Express web app in the root of our project once again and change the name of the app to Froggyweb. Next, I'll install our module by referencing it directly using a file system reference. You'll notice that it installed everything just fine, but I don't have a directory reference here. It's just the name of the module. This clearly will not work for us the way we want. If I was to use NPM update, this module would be overwritten by the module registered in the NPM registry using the name Membership, and there happens to be a module called Membership up there. If you're working on a team, and you don't want things like this to happen, be sure to use a Git or a Github reference here. I can update this manually by installing it again if I want to. And that's what I'll do for this demo. Next, I'll need to have a bit of middleware that will sense when a user is logged in and control the session bits, etc. And for this, most Node developers use something like Passport. I'll install that using NPM. Passport is fascinating, and if you're a .NET developer, you can think of it like forms authentication. Forms auth doesn't handle your persistence or the notion of a user. You just tell it to log someone in and out and whether to persist the session. Same deal with Passport. If you're a Rails developer, this is where Node and Rails part ways in terms of philosophy. Devise in Rails fashion does everything for you. It bolts itself onto Rails, creating routs, views, controllers, and so on. Passport, on the other hand, is a single piece to the puzzle. Our Membership module is the other piece. The Passport documentation is pretty good, but it can be a bit of work to figure out just what to do, so let's go through this step by step. I've just installed the Passport module. Now I need to wire it up to our application. The first thing to do is to set Passport's strategy. You can have it authenticate using external services like Facebook, Twitter, Google, so on, using OAuth or OpenID. You can use basic authentication. Or you can use what they call a local strategy, which basically means you'll tell Passport what to do. For that, I need to create a local strategy. And the first function it wants from you is a way to log a user in, eventually returning an error or the user record. What you see here is the Passport template I pasted directly from their site. Let's comment that out and, in its place, I'll use our new Membership module. (Typing) That was pretty simple. Again, this is the function that Passport will use to log the user in. All we need to do is fire off the Done callback. Notice also that this function accepts an email, password, and a callback. This is why I wanted to keep my API call the way I did. Even though I've suggested that you pass objects instead of primitives, it keeps it in line with Passport, which I think is important. Okay, we're almost there. Now I need to tell my application to actually use Passport. The first step is to provide the cookie parser with a secret key. Next, I'll initialize Passport and pass that Passport instance to Express as middleware. Finally, I'll plug Passport's Session Handler into Express. Okay, let's run things and see where we're at. I can do that with Express by calling Node app.js. I forgot to install the dependencies for Express. As I mentioned before, when you clone Github repositories or generate Node apps, the Node modules won't be there. You'll have to install the dependencies yourself using NPM install-d. Now that that's done, let's run our app. And, no problems. Hey. Let's fast forward a bit here. Added some styling to this site using Bootstrap and put our login page right on the home page just for simplicity. Now I just need to plug in a login rout. I'll remove the user.js rout, that doesn't make much sense, and I'll add an account.js page to the routs. Here's a handy tip I picked up from Geoffrey Grosenbach of Peepcode. When you work with routing, as we're doing here, just pass the entire app in the constructor. That way, I can wire the routs in one place as I need. (Typing) I'll require Passport here and set up some redirects that I'll use with Passport in just a second. Next, I'll create a post rout for accepting login credentials, and I'll hand it off to Passport's Authenticate method along with the redirect manifest. And that's it. If this looks weird to you, it does take some getting used to. Express works with middleware in a very elegant way. You can chain whatever functions you want into the Rout Definition function, as long as the final argument is a callback that tells the response what to do. All that Express cares about is getting a callback at some point, and it will look for that as the last argument of the rout registration. This pluggable model is what makes Express such a joy to use. But if you're not used to it, well, it can be a bit confusing. The final thing to do is require our accounts rout so that our new rout is picked up in our apps configuration. I'll be sure to invoke the module and pass in our app object. Okay. Let's try and login and see what's what. I can use the credentials I've been testing with, and an error. Believe it or not, this is promising. Passport doesn't know what to do to remember who the user is, and that's because we never told it what to do. We do this by calling Passport.serializeUser and passing in the function we want used. This function is called after Authenticate. Whatever user record is returned from Authenticate is passed here. What we need to do is save some information into this session so we know which user to pull out later. And I'll do that by serializing the user's authentication token. Next, I'll just need to de-serialize the user so Passport can pull the user back out. This first argument passed in is the thing I asked Passport to serialize, which in our case is the authentication token. I'll use that to fetch the user out of the database and then pass the callback directly. Okay. Let's try this out. Logging in. No error, which is great, but how do we know if we're logged in? Let's head over to the index rout and pass user information to the view. When you login with Passport, the De-serialize User function will take the user record you provide and will stick it on Express's Request object for you to use anywhere. So here, I can just send request.user down to the view. One thing I'm not showing here, which is very important, is that if you want to protect a given rout, you need to plug a global function into the Rout Definition. You can do this by flexing Express's extensible middleware approach. Here's a typical function that I grab from the Passport Github repository, ensureAuthenticated. You can plug this into your Rout Definition function right after the Rout Definition itself, and it will get called inline. If everything's okay, then the next function will be called. If not, the user will be redirected. Okay, let's head over to our view page and output our user's email to see if he's there. And, yes! Looks like we have the beginnings of a pretty neat system.
Summary and Goodbye
Developing Node apps can be frustrating, exciting, exhausting, and exhilarating. You can have all of those feelings within the same 20-minute coding period. If you work with Node and find yourself feeling frustrated and exhausted, remember that you're likely working in a system that's alien to you as a developer. If, like me, you come from a procedural, object-oriented background, making the transition to Node will take patience and effort. Also, knowledge of various patterns, which is why you're watching this video. These are the patterns I've learned after working with Node for the last year and a half. As I mentioned, I'm not a Node expert, but I have been programming and working with the web for a very long time, and these are the patterns that I find most helpful. And they keep me from getting overwhelmed and exhausted. I do hope you've enjoyed watching this production. I'm going to keep working on this module and others. And if you have any feedback, I'd love to hear it. You can drop me at line at the email onscreen or drop me a note on twitter. Thanks again for watching. See you soon.
Rob Conery co-founded Tekpub and created This Developer's Life. He is an author, speaker, and sometimes a little bit opinionated.
Released11 Jul 2014