What do you want to learn? Leverged jhuang@tampa.cgsinc.com Skip to main content Pluralsight uses cookies.Learn more about your privacy Large Scale JavaScript on Client and Server by Shawn Wildermuth JavaScript is an important language to modern web development. Most examples are trivial in size, but many developers need to work in large scale projects. In this course I'll show you the techniques in dealing with building large JavaScript... Start CourseBookmarkAdd to Channel Table of contents Description Transcript Exercise files Discussion Learning Check Recommended What is Large Scale JavaScript Introduction Welcome to the first module of the Large Scale JavaScript on Client and Server course. My name is Shawn Wildermuth. In this module we're going to be talking about what is large scale JavaScript and what are the problems we're trying to solve. This is going to include how large is large JavaScript, what is wrong with JavaScript, the problem of DOM-Centric Development. I'll talk about libraries versus frameworks in JavaScript, the impact of rich JavaScript applications, and then we'll talk about some of the solutions, including how Meta-Languages can help us solve some of these problems. Let's get started. How Large is Large? One of the first questions you'll need to ask yourself is whether your application is large enough to worry about scalability. When we talk about size we're talking about three different things. The number of lines of code, the size of a team building that code, and the amount of complexity in that code. The hint here is your application should be large enough in any of these three areas to take some of the approaches we're going to talk about. This includes lines of code obviously, and typically we're talking about lines of code in the range of 10K or more, that these are very arbitrary sizes, but also see that the size of a team, a team made up of a number of members can really add to the need to scale up your JavaScript even if the size of your code is small, and if you're using six or more modules in your application, and these are logical modules, six different areas of functionality in your application, then it's probably complex enough to be deemed as a large JavaScript application. Now if you're in a startup and you've got a two page quickie website that just needs to go up and collect information, let's say email addresses, this probably doesn't qualify as a large JavaScript application, but most applications that I look at that are in the Enterprise tend to be a large JavaScript application because one of these three, and in many cases all three of these are found to be true. The reason is that there is often enough complexity in Enterprise JavaScript applications to not only have the amount of modules you need, but also have a larger team size in of course the excess of number of lines of code that you've written yourself. Even if you're application doesn't qualify as a large JavaScript app many of these techniques can be useful because we're talking about making code that is going to scale up to the number of people that are working on it, the number of users that are using it, and the amount of complexity that needs to be managed. This doesn't mean that every website out there should take on a lot of the techniques in here, but in my personal opinion most JavaScript applications, anything that can be called an application versus a website, should probably look at a lot of these techniques. The goal here is to take anything that you're dealing with in JavaScript and look at it as something that is complex and attempt to make it simple. This is a quote I really like that sort of comes down to this point and that is you're trying to make things simple, but not simpler than they need to be, and so the idea of having hundreds of small modules that are all interconnected may make you feel better about yourself or make you feel like you're an architect, but it doesn't mean it's the right approach. The problem is when we build JavaScript applications we have our hopes and dreams wrapped up in it. We intend, as designers of applications, as developers of applications, to build these great buildings of code. We call them architecture for a good reason because we're looking at the way we design and build the systems of a building, whether that be a small shack or whether that be a large high rise. You approach it in many of the same ways. You're trying to figure out what are the components required here, and it's not just eat through the apartment and a building, but it's also how the plumbing system works together across all of the apartments, how electricity and air conditioning systems can cool the entire apartment building, how different parts of your solution are connected by things like elevators and stairs, until you have got these components, but also ways these components are related to each other and share different sets of services. The problem is at the end of the day we start with these great, grand plans and schemes and blueprints, and we often end up with something that looks sort of glommed together because things like schedule pressure, churn in team members, team members leaving and joining the group, as well as unexpected changes to an application end up having our buildings look a lot like this, where we're sort of attaching things ad hawk. If we were constructing the building from scratch and we just started coding we're going to get a building like this. The hope is that if you have enough foresight to know where you're going and that you're moving in small enough sprints that you can avoid ending up with web applications that look and feel like this when you look at the code base. What is wrong with JavaScript? The problem as JavaScript developers or web developers is that what you're building is based on the JavaScript language. JavaScript simply isn't designed for the sort of scalability problems you're going to run into. It is, in fact, a scripting language and as many of you already know this means the language wasn't really built with the idea of composition, and modularity, and other things that are going to be key to you building scalable JavaScript. This doesn't mean JavaScript isn't up to the task, it means that you have to understand the language better than you did other fourth-generation languages. If you've come from Java, or C#, or C++ the languages themselves were built with the idea of information hiding, and object oriented programming practices, and rigorousness of design, JavaScript wasn't. We can imply those techniques on top, but we don't get them for free. The problem is that JavaScript as it stands now, before EcmaScript 6.0, it's missing some key features. There is no sense of packaging or namespacing. It's too easy to use the global namespace to simply pollute an area with your own code and see it collide with other code in that same namespace. It includes prototypical inheritance, which is a useful technique, but one that is very different from many other OOP languages. Doesn't mean that JavaScript isn't object oriented, it simply means it's object oriented in a different way and that we, as developers, have to learn that way to do these things in the right ways. In most cases the tooling for JavaScript is simply too immature. It doesn't pass the same level of muster as other language IDEs and other tooling out there, including profiling tools, debugging tools, etc. We've got the browser tools, which are great. We've got some integrated environments like WebStorm, like Visual Studio, but none of them are quite as mature as the other nonscripting environments are. Unfortunately, JavaScript is ubiquitous and so it makes it a really promising language to build the stuff in, but we have to, as developers, realize that we don't have the proper tooling or the proper language features to make that work. This lack of rigorousness leaves us in a bad place. It means that when we start developing in JavaScript we're shown this sort of code a lot. We're using things like jQuery and other libraries to ease the development, and certainly been a boon to web development, but we start, we say okay we're going to be handling some callbacks, we're going to be handling click of certain elements in the page by manipulating the DOM directly, but very quickly, we end up with these monstrosity of JavaScript files that include, many times, thousands of unrelated business logic, as well as UI logic. We also get into this place that many people call Callback Hell, and that is again, you're starting with the same idea where you're handling event callbacks, like this click event, and then that ends up in these great nested morasses of callbacks where one callback begets another callback, begets another callback, and the callback systems are very useful in that it ensures that your application is going to be nice and asynchronous, which means the UI is going to continue to be fluid, but it does mean that writing the code, writing the code in the way that most people will see examples out on the web, are going to look like this, and this is very unmaintainable code. Even with comments, the sheer nesting of this is going to make it hard to read as different developers take over that code, or need to add features, or fix bugs in this code it's going to add a lot of cost to that. When we get into thousands of lines that look like this we're not only dealing with a large codebase that we need to figure out how to scale up, we have a large codebase that�s hard to conceptualize and maintain, because as we all know, developers don't tend to stay in one job for their entire life. Someone that wrote this code invariably is not going to own this code, is not going to be there when this code needs to be maintained in two years, and three years, and five years. Frameworks and Libraries As we're building these large JavaScript applications we have the idea of frameworks and libraries, and in some ways we interchangeably use these phrases, and before we get started in building and describing the solutions, I want to really differentiate these two ideas. They really aren't the same. When I think of a library I think of it as a toolbox. It's simply a set of things I can use to build something. It isn't the structure of what I'm going to build, a hammer doesn't say that I need to build a 2x4, connect it on four sides in order to create a frame for a door, but it does say that I hammer nails, has a very specific purpose, and interacts with other components in a very specific way, but it doesn't imply what we're building. Frameworks, on the other hand, are going to imply some practices in the way we build our codebases. Frameworks, in many ways, are going to imply some of the patterns we need. When someone is building a home they may decide they're going to build an A-frame home or they may decide to build a boxy apartment building. These designs have certain common metaphors that are going to be repeated and frameworks are going to try to teach you some of those metaphors that are going to work best with it. All the techniques we're going to talk about later in this course are going to work regardless of which of the major frameworks you're going to be working with. Before that, our most common out there are KnockoutJS, AngularJS, Backbone, and EmberJS. All of these have the facilities that are going to allow us to create large scalable codebases. One isn't necessarily better than the other. In many ways it's a stylistic choice. Angular is often the one stop shop when you want someone to really dictate how you're going to create a variety of different sets of architectures, whereas Backbone is much like Angular where it supplies a lot of what you're looking for, but it wants to take a lot of the things like data binding outside of the code and use things like views in JavaScript. Knockout, on the other hand is purely about data binding alone, and so if you want to sort of pick the different tools you're going to use Knockout's a good one that says hey I do data binding, and then you're going to go someplace else to get things like routing, and composition, and dependency injection, those sorts of things. So simply put, you can build scalable JavaScript by using a framework, but a framework isn't responsible for it. Using a framework can certainly help you build scalable JavaScript easier, but if you're in a shop that has specific needs, and none of the frameworks meet that, hopefully in this course you will learn the techniques that you will need to implement yourself instead of having to rely on a framework to help you implement them. The Solution The solution that I propose is really built under three main concept tiers and we're going to break these different tiers out in different modules and talk about them separately because each of them contains a number of different tools and different techniques you're going to want to learn. These are going to start with the idea of maintainable code, writing code that you know will need to be edited and changed by not only you, but others on your team. You also want code that is scalable, so that on a particular page in a website not every line of code is loaded into the JavaScript interpreter. Testable, you need to have code that you can write tests against to make sure that the quality of that code is high, but also that changes to that code don't break older code. This whole idea of progression testing becomes important in large codebases regardless of language and JavaScript is certainly no stranger to that. When we talk about maintainable we talk about a couple of concepts. The idea of modularization of code and to separate those concerns into modules. Though these concerns may be actual things like modules they may be things like classes, but we're talking about lumps of code that have a single responsibility. In terms of scalable, we need code that is composable, so that we can attach code and services to the rest of the system as necessary, again, so that we're not dealing with one giant lump of code that works on every page and that is going to slow down simpler pages. We also want it to be loosely coupled so that any code that you're running doesn't have a hard dependency on another piece of code. In order for your code to be testable we're talking about you encapsulating functionality and then exposing facades that you can write tests against. While you can certainly just do testing through UI testing using beta testers, using a QA team to test that, your code quality is going to be better, and this is much more important for large scale JavaScript that you're running unit tests against it. The idea here is that you'll run unit tests against the facade of your functionality, so that even as that functionality changes under the covers as new features are introduced or bugs that are fixed, those same unit tests will be run so that regression tests are going to be valid. This idea of regression becomes really important when you're worried about the quality of your code because you want to make sure that new features, bug fixes or any other sort of maintenance you're going to do doesn't break old code. What about Single Page Applications? In the last couple of years this idea of Single Page Applications or SPA has really come to be a vibrant force in the JavaScript community and I'm often asked do I need to worry about scalability, I'm using Single Page Applications, therefore shouldn't this just solve scalability for me? There not really that interrelated. You can write bad Single Page Applications just like you could write bad backend solutions before. Unfortunately, I think the idea of Single Page Applications is actually a problem. I think that Single Page Applications are actually a misnomer. Your applications should include as many pages as your application actually needs. The idea that we're going to try to shove everything into a thick client that happens to be written in JavaScript actually hurts scalability. The reason is you're asking the browser, and any browser, mobile browsers, older browsers to take in all this JavaScript and do the right thing in these different situations, and JavaScript wasn't really built for that. In many ways the scalability problem has been made worse by Single Page Application, in fact, in many ways the Single Page Application mantra has made the situation worse because we're shoving more JavaScript into individual pages. When you're very rigorous about what you do this can really work and work well. I'm not talking people out of using SPA or Single Page Application philosophy. What I am saying is that the more JavaScript you're going to run in a single page the more this is important. We used to be able to get away with bad JavaScript before because every time we had an action it would go back to the server and re-render the page and reparse all that JavaScript. Now that we're shoving a lot of JavaScript we're getting to the edge of what the browser can handle as a single giant set of JavaScript. Now remember, we're talking about the browser parsing your JavaScript in every library you're using. The amount of JavaScript we're asking it to read tends to be much larger than it ever was and certainly browsers are more capable of doing that then they ever were, that doesn't mean we need to use every cycle that is granted us by the new browser engines. I'm going to ask you in this course to learn to be rigorous. Because I truly believe Single Page Application is a misnomer and I would like to get people out of the idea that their entire web application or website needs to be a single page. I'm going to, in this course, call them Rich JavaScript Applications instead. The idea is the same. You're going to be building thicker clients in the website using JavaScript and JavaScript frameworks and libraries, but I'm trying to get us away from the implied nature of there needing to be an actual single page. Typically rich JavaScript applications have a couple of common features. They're going to encourage loose coupling so that they can work well together as those applications grow. They're going to be focused on using data binding and REST instead of using direct DOM manipulation and coupling of business logic with DOM manipulation directly. They tend to be longer-lived pages, but we may have a number of these for different parts of our application, and that typically they're going to use one of the MV* models, whether that be model view controller, model view view model, model view whatever you want to call it. It's going to use one of these philosophies to keep the responsibilities separated. Meta-Languages The last thing I want to talk about in this module is the idea of Meta languages and I put quotes around this on purpose because the first of these meta languages we'll talk about is actually just JavaScript. EcmaScript 6, which is the next version of JavaScript that's coming, is going to solve some of these problems for us and I'll show you how. They're going to introduce different facilities to the language that will allow us to handle things like modularity, like separation of actual classes, like inheritance in much simpler ways than prototypes. We may not get to EcmaScript 6 in all browsers for quite a while, but I want you to see what's coming down the pike. Once we see that, we're going to also look at the three meta languages that are in use today, CoffeeScript, TypeScript, and Dart. These are three languages that compile down into JavaScript and they have some different philosophies around how they work. EcmaScript is going to add features like modules, and classes, and inheritance into the language of JavaScript and here's a simple example of a class in EcmaScript 6. You can see we have a class name, we can see that it extends something, we have the idea of a constructor, and we have the idea of members, both data getter setters, as well as simple variables. All of this can be done in JavaScript today, but we've got to understand how to make JavaScript pretend that it works, and we'll be showing you that through the rest of the course. CoffeeScript is one of the early champions of let's not write the JavaScript directly, let's go ahead and write it in another form that compiles down or generates our JavaScript for us and they use a much simpler syntax. It doesn't look and feel like JavaScript, and so learning CoffeeScript does require you actually learn a different syntax, but they have the same ideas the EcmaScript is bringing into the language, things like adding keywords that then generate the correct JavaScript behind the scene. This works now so you can write your code in a structured way and then have the JavaScript that is necessary to have these different implementations be generated for you. Of course, the problem here is that you need to understand the generated JavaScript since that's what you, in most cases, will end up debugging. Another meta language out there is TypeScript and TypeScript is interesting because it takes a different approach than CoffeeScript. Instead of inventing a new syntax or basing a new syntax on something other than JavaScript, TypeScript is a superset of JavaScript. It simply adds on top of JavaScript some of these additional Ecma 6 pieces of functionality, and in many ways TypeScript is a way to write your JavaScript today knowing that it will actually move into EcmaScript 6 when that is in fact possible. In taking features in TypeScript, Microsoft, who produced and open sourced it, is taking their feature set directly from EcmaScript 6, so if it's called a module in EcmaScript 6 it's called a module in TypeScript etc. Here we can see that the code here looks remarkably similar to EcmaScript 6 with one major difference and that is TypeScript brings in this idea of type safety into the language as well. Now this type safety isn't enforced at Runtime, but it is enforced by tooling at compile time, so you can find out that you're having a mismatch of data types, where that is valuable, and in large codebases that can be really valuable. Dart, which is Google's attempt at another language for the web, and also compiles down to JavaScript, does the same sort of idea. Uses a little different syntax, but it's supposed to look and feel a lot like JavaScript. Problem with these meta languages is they don't get us quite there. None of them are perfect. None of them are going to solve all of our scalability problems, and so while you can use them to implement a lot of the things we're going to talk about, they're not going to solve anything out of the box for you. So for the balance of the course I'm actually going to be focusing on how you do this in pure JavaScript, and if you happen to be using one of the meta languages, or you're viewing this course late enough that EcmaScript has made itself into every major browser, then you can just use those facilities instead of the pure JavaScript solutions I'm going to be talking about. Summary Let's wrap up this module and start to talk about the pragmatic piece of scaling your JavaScript out. What we've learned in this module is that the idea of large scale tends to be subjective and that you want to use good techniques regardless of the kind of JavaScript you're writing. Now there are cases where you don't want to bother with some of the ceremony we're going to show and introduce, but in many cases you can build better maintainable, scalable, and testable code by using these techniques. Simply writing JavaScript in any way you want isn't going to be the challenge of handling these large scale problems. We're going to ask you to be rigorous about the way you write your JavaScript. These good development practices will solve a lot of the failings of JavaScript and allow you to build rich, mature, high quality codebases using nothing but JavaScript, and like I've said, the three keys here that I want you to focus on is the idea of creating JavaScript that's maintainable, scalable, and testable. Both frameworks and meta languages can certainly help here, but they have to be backed by good developers and good developer practices, and the goal of the course is to make you a better developer, and when it comes to JavaScript enlarge JavaScript scenarios. This has been module one of the Large Scale JavaScript Course. My name is Shawn Wildermuth of Wilder Minds. Thank you. Maintainable JavaScript Introduction Welcome to the second module of the Large Scale JavaScript on Client and Server course. My name is Shawn Wildermuth of Wilder Minds. In this module we're going to be talking about maintainable JavaScript. We're going to start out by talking about some application frameworks and how they empower maintainable JavaScript. We're then going to dive into some tasks including avoiding the Global Scope, using strictness in JavaScript, modular JavaScript, dependency management, smart asynchrony, and finally, loose coupling. Let's get started. Application Frameworks In this module we're going to focus on writing JavaScript that is maintainable. The important lessons I'm hoping you're going to learn is that building large scale JavaScript projects requires you to be more consistent in the way you write your JavaScript. This doesn't mean that for small projects doing things in a quick and easy way isn't still a good approach, but when you need the rigorousness of a large scale project you're going to need to be a lot more careful. There's an old adage and I'm not sure the numbers match the way we write software today, but it used to be that for every dollar we spent in writing software $0.90 was spent in maintaining that software. Only $0.10 out of that dollar was spent in writing the original project and a lot of what we're going to focus on are techniques to make sure that maintaining that JavaScript is going to be as easy as possible because at the end of the day the mechanic working on your code may not be you, may not be your coworker that was with you while you wrote the code. Very likely it's going to be someone that is at your job or maybe even in your job when this code needs to be maintained, changed, updated, features added, and so to be good developers we want to make sure that our code is readable. Now this used to mean adding lots and lots of code comment and this is becoming less important. We're going to find out in this module that there's a number of techniques that will help us build better, smaller units of work of code, so that when someone has to take over and look at what we're doing code comments will help, but the structure and simplicity of the code is its own reward. It makes it easier to read, easier to maintain, and most importantly, easy to understand. Now many of you that are watching this course are already invested in some frameworks out there. These frameworks can help you build large scale systems not only quicker, but with more rigorousness. These techniques for writing maintainable JavaScript are going to work regardless of which framework you're going to use and some of the examples we're going to talk about specific libraries and specific frameworks, but I want you to focus on the kinds of techniques not the specific code examples. If you're writing large scale JavaScript and you're using AngularJS, Backbone, EmberJS, Knockout, and other associated libraries like Sammy using Durandal or even using EcmaScript 6, and a lot of its features that are going to support this more rigorous way of writing JavaScript, you're going to be able to apply these lessons to any of those application frameworks. We're not here to tell you, oh in order to make this scalable you have to use Angular, or Backbone, or Ember or one of these in particular, because the techniques are all going to be important lessons for you to figure out which framework fits best with the kind of people you have building your applications. In many ways, choosing any of these is not going to be the wrong option. Many of these have enough parody in feature set that it won't impact the scalability of your JavaScript if you use any of these frameworks. You may find that any of these frameworks may appeal to you for reasons that have nothing to do with scalability of your JavaScript. At the end of the day these application frameworks can encourage you to write better code, but ultimately it's your responsibility. You the developer are going to have to make the choice to be rigorous and careful about how you build code so that the resulting project is highly maintainable, therefore it's going to be easier to test, it's going to be easier to scale, and the quality of the code you're going to produce is ultimately going to be better. Global Scope In building maintainable JavaScript a great place to start is with the global scope. The idea behind the global scope is that JavaScript doesn't dictate that you have to write JavaScript inside classes or modules or anything like that. You can simply write it one line after another because this is a scripting language at the end of the day. One of these techniques that will help you write more maintainable JavaScript is to avoid using the global scope. This doesn't mean you can never write a global object, but you're going to want to minimize the number of global objects, often only having one in your entire application. The problem is that the global scope is really easy to pollute, so that you can create many variables, and managing these variables, and who reads and writes to them, and protecting them from people polluting your variables becomes a full time job. You also have the problem with collision. One part of your application may decide to use a global variable called customers and someone else later defines a new variable called customers in the global space, and suddenly it means something different in different spaces or you're simply destroying a copy of those customers for this other piece of code, and finding, nailing down those overriding behaviors tends to be a long drawn out process. You want to avoid the global scope to limit the risk of overwriting these existing variables. Now these variables may exist in your code or libraries you're using. Just because you're using libraries out there, whether they're jQuery extensions, whether they're utility libraries, you can't control what global variables they're going to use, so if you limit the number of global variables you're using, the chance that you're accidentally colliding with them becomes pretty small. The other problem with global scope is that it tends to encourage us to write large blocks of code. When we start to deal with global scoped variables we start to see that projects that use a lot of global variables end up being projects with two or three giant 5,000, 10,000 line JavaScript projects where they're simply just throwing code at a problem not having a sense of how to format and how to structure the code so that it's easier to maintain. The technique here is to hide your code from the global scope so you don't accidentally create global variables. What you want to do is use the idea of a function scope. In JavaScript the function scope is the one scoping mechanism we have. There isn't any such thing as block scope in JavaScript, there's only function scope. We need to put our code inside of a function in order to protect it from the global scope. The technique here is something I'm going to refer to as the Self-Executing Anonymous Function. This same technique has a number of names including the Self-Invoking Anonymous Function, and the Immediately Invoked Function Expression. All of these are interchangeable for the same idea. I'm going to refer to it as the Self-Executing Anonymous Function. Let's see what it looks like. The idea here is that you're defining a function that is going to be executed as the file is parsed. You're not defining a function you're going to call later, this is simply a piece of code that's going to execute, and because we're using a function to do that execution we're defining a scope here so that variables created here aren't going to be in the global scope. If you're not familiar with these, let me just walk through this briefly. We can see that the function itself is surrounded by a set of parentheses and the reason for that is it is going to enclose it into a function object and then the last two parentheses tells the parser to execute that function object, so this code gets executed during parsing. That's an important concept to get in your head. If we create data, variables, functions inside this function scope it's not going to leak to the global scope. This variable doesn't exist in the global scope it is only visible within this function scope. If we decide to write the code that we need to execute on a particular page or in a particular context, like here is some jQuery code, we can use this cached object because we are using a closure to ensure that we have access to that without it having to leak out into the global scope. This idea of enclosing your objects inside one of these self-executing anonymous functions is a simple way to protect you from creating new, accidental global objects. You can pass data or context to this function scope. You do this by specifying the data as parameters to this function, so here we're passing in the Global jQuery object and we're doing this to prevent the lookup of the Global object called jQuery. Our code inside here is just going to be passed in as a parameter and then used within this scope. You'll see this pattern in a lot of examples out there. You can add other variables to this pattern in order to add data that's being passed into this function that's being executed. So if you collapse everything before those last parentheses, you could think of it as a simple function object. The fact that you have code in here that's protecting it from the global scope is a side effect of it. In most cases, I'm going to support that much of your code, even if you're using a framework like Angular, or Ember, or Backbone you're still going to want to protect it by using these self-executing anonymous functions to prevent accidents with global variables that may overwrite other global variables that other libraries or even your code need. This is the belt and suspenders that protects us from accidental creation of global objects. Strictness in JavaScript If you're writing JavaScript today, I would hope that in most instances you're going to turn on strictness. In recent versions of EcmaScript you can opt into using a more strict definition of what JavaScript is. This is going to enforce the best parts of JavaScript. It's going to disallow things like the automatic creation of variables on first use. It's going to enforce type, so if you create an object as an integer you can't store something like a string in that type, and a number of other things like read only and write only access to objects. This is going to throw exceptions on these bad practices so that you're going to show these errors earlier in your development cycle. If you use bad practices in JavaScript, using strictness will tell the Runtime to throw an exception when it finds one of these cases. This means that the quality of your code is going to be better, therefore maintainable, and is going to bring those errors to the developer earlier on so they can be fixed when they're cheap to fix, not later when you have to find one of these bad practices are in production code, or are in code during acceptance testing or somewhere later down the line. You want these sorts of simple problems to be found during the development phase not during the testing phase. While strictness is really useful, it's not a replacement for something like JSLint that is going to do some code analysis as well. Ordinarily in JavaScript if you simply create a new variable without the var keyword and assign it some value this just works fine. If I create a variable like we're seeing here with the y, and creating it as a string and then storing a different type in it, that also works fine. There are a number of these bad practices in JavaScript that are allowed because this is a simple scripting language. As the language has matured we've decided that there are some things that are bad practices. Now the problem is JavaScript runs on so many devices that not every device is using a modern version of JavaScript, so we have to have a way to enable strictness without breaking the old browsers. The way we do this is to add the string use strict in quotes, followed by a semi colon, so that the parser knows to turn on strictness within this scope. Now notice that I'm using use strict here within one of these self-executing anonymous functions. This tells them that only the code within this scope should be strict. Now we could do this at the global level as well, but we may be using libraries that are depending on the old behaviour, so the general practice here is to only do this for your code, don't try to accidentally enforce it for any libraries you're using. In this case, because strict is on, we're going to see that the x=0 is going to throw an exception because it's an undeclared variable. We have to declare it with a var for it to work. In addition, the y variable being created as a string when we try to assign it a non-string it's going to complain because of the type change, and again, there are a number of these, I'll let you look up what they are, but the best practice here is to use strictness in your own projects. Modular JavaScript When you're building a large scale project one of the keys you're going to want to focus on is making sure that project is modular. You want small, discreet units of work that are going to be put together to create a larger piece. You can think of this as building a building with a lot of small pieces that all sticks together. You can think about this in the way we build PCs with small pieces that all make up a larger unit that knows what it's doing or even something like the space station that is made up of all these little modules that stick together. The trick with modular JavaScript is that it's going to be more maintainable because you have smaller units of work that have on job and one job only. When we talk about a module we are talking about a simple part that is an individual unit that is used to create a larger piece or a larger structure. In our case this is code. For us, we should focus on that an individual module is a single unit of work. This unit of work may be vertical or horizontal and what I mean by that is, a vertical module might have a very specific purpose. Show the date or show the login on individual pages. It has a small unit of work that knows the operations it needs to do to do a single task and this may be, let's say in the case of a website that is a dashboard, that you might have a module for displaying a chart for the stock price, a chart for the amount of downtime in the system. Those are vertical because they have simple operations and they belong on specific pages. Horizontal modules are ones that are crosscutting. They cross each of these vertical cases and can supply some work to a number of different modules. We can think of these as logging, and data access, and security as these crosscutting types of modules. They're modules that may be used by other modules to get the same sort of simple behaviour for a lot of different pieces of work, but the thing to focus on here is that each module needs to have a single unit of work. If you're building a module and you're being asked to add a feature that is unrelated most likely you need to build another module. Modules are typically small, reusable, individually testable, loosely coupled, so that you don't have modules that have a strong relationship with another module, and they typically are discreet. When we're building JavaScript modules I like to think of certain rules, like that there is no DOM manipulation outside of an individual module. If a module needs to interact with the HTML DOM you're not going to try to do that code in a helper function or in some global function that you're going to call. You want all that DOM manipulation to be inside of a module. That module should have little or no global declarations. Often, a module is going to be registered with some global object that has a collection of these modules, but in itself it shouldn't create its own global declarations. You should not hard couple one module to another. If you have two modules that are related on a page they need to have a way to communicate with each other that doesn't require one to have a reference to the other, and I'll show you some ways to do that in a little bit. Then finally, not only shouldn't you declare global objects, you should not access global objects. If a module needs a service or another module it should be passed into that module. Let's look at some module patterns across a couple of different libraries including simple JavaScript. Now the module pattern, this is a fairly common pattern in JavaScript, it's using a self executing anonymous function to build up an object that is returned. In this case, it's building up a set of code, it may be number of functions, it may be some data, that returns a singleton that represents the module, so this module as it's being passed to different modules will be able to call the fillCache or access the data that is in the actual cache here. This is the standard module pattern and there's some other courses here at Pluralsight that talk about this more in depth, the module pattern in JavaScript. Another modular pattern in JavaScript is the class. Now of course there aren't classes in JavaScript yet, but there is this notion of a constructible object. This is typically done with an initial capitaled named function and in this case when we create a new instance of an animal using the new keyword this will create the object and then we can add to the type of animal its own functions. This pattern is really useful when you need to create non-singleton implementations, when you need to have a number of a certain type that you're going to create a collection of employees or customers or invoices. This is typically where you can keep all that data encapsulated in a single place. It's still modular JavaScript even though you're creating more than one instance of these objects. In Angular they have a different notion of the idea of modular angular. It can get a little confusing because we have this notion of creating a module and a module is really a collection of related code. We can think of an angular module as being a modular pattern, but it goes a little bit deeper than that because from that module we can create different types of modules that we can reference. We can create modular pieces that all interact with each other. Here's an example of a module adding a controller to it, a specialized type of code that is executed, that's going to include any code that is going to be required to interact with some part of, usually, a view inside of HTML. AngularJS also has this notion of services, and factories, and other sorts of objects that are created and created once and passed in to other pieces of modular code. In this case, we're creating an object called a dataFactory that contains all of the operations and data that we may need in different parts of Angular. In this way, AngularJS supports a handful, not just these two types, but a handful of different kinds of modular code to put together a larger AngularJS application. When we think about the different frameworks out there they handle the modularity in different ways. Plain JavaScript has the notion of things like namespaces, the module pattern, the class pattern, but we can also see in AngularJS there are modules, services, factories, controllers, directives, and even more different kinds of modularity. In Backbone namespaces and objects become the modular patterns, though by using Marionette with Backbone you can get the notion of full modularity inside Backbone projects. EmberJS does this with extending built-in objects much like Backbone does and it still can use the EcmaScript 6 and plain JavaScript module modularity we can see above. Durandal uses asynchronous module definition, which we'll talk about in the next video for defining these modules, and then EcmaScript 6, when it releases and it is supported by all the browsers, is going to be using something called CommonJS, which is a standard for building modules in JavaScript. Ultimately, most of these are going to end up in the CommonJS compatible format that's a standard that most of these are going to end up adopting because then we'll have one way to define modules across the different frameworks. Dependency Management Most software has the notion of dependencies. A dependency is simply a module that other modules rely on. When we think about the way we structure our projects we may have one type of module that depends on other types of modules, and they may in turn depend on their own set of modules, and so walking this tree of dependent objects and knowing how to construct them and pass them in is the purpose of dependency management. Using dependency management is important when you're building large scale JavaScript applications. When we talk about dependency management we're really talking about a system of handling these dependencies across an application. This is usually referred to as Dependency Injection or also sometimes referred to as Inversion of Control. This is going to allow us to pass dependencies into modules as they're defined without depending on global scoped variables, which is typically the way this is handled without dependency management. When we're requiring a dependency the dependencies that they depend on are handled so these cascading dependencies are fulfilled for us. There are really three ways that we can handle dependency management. First is something called RequireJS. This is using something called asynchronous module definition and we'll talk about this in a moment. CommonJS is a new standard for defining these dependencies and we'll see what that looks like as well, and then AngularJS has their own way of dealing with dependencies, so I want to show you that as well. In this case, we're using Asynchronous Module Definition and we're using a global function called define and this is something that RequireJS, as well as some other libraries, supply you to define your modules, and by default in RequireJS this define method is going to assume the name of the module based on the name of the JavaScript file, and then here we're using the simple module pattern to define the object that is going to be this module. This empty array is a way to pass in the set of dependencies that this module needs. In this way I'm asking for the objects that I depend on and then the people who require me are going to ask for the objects that they depend on. They don't need to know about my dependencies they just need to know that they depend on me and the system, the Asynchronous Module Definition here is going to determine how to fulfill all these requests and cascade or chain these dependency requests in order to do it properly. We can also specify a name for the module if the name of the JavaScript file isn't sufficient enough. On the other side of this is being able to use dependencies. The require function is the other piece of the Asynchronous Module Definition in that you're saying I have some code that requires some dependencies and I'm going to give you a list of those dependencies that I need. Again, I'm not going to tell you what dependencies each of these dependencies need, they're going to define that themselves using define, and then once require has gathered up those dependencies, and is ready to tell you you can load your code, it's going to provide these dependencies via the callback function. It's going to map each of those dependencies into an input parameter into the callback function then you can use these dependencies in any way you need. These dependencies may simply be pointing at specific folders or JavaScript files. The CommonJS specification is similar to the Asynchronous Module Definition, but it assumes that all of the code is readily available. The CommonJS spec is used in both client-side code that wants to deal with dependencies, as well as server-side code. NodeJS by default includes an implementation of the common JS spec, so that you can use the same sort of dependency management on the server, as well as the client. Much like AMD, the CommonJS spec says I'm going to have a function called require, but in this case the require is just going to say go get this required object and pass back that object so that I can use the pieces. It doesn't require that you structure your code to include all the requirements, but allows you on a variable by variable basis to go ahead and grab or find an individual requirement. On the other side of this, inside of that API JavaScript file there's going to be a global object called exports, and this exports is going to allow you to add properties to the exports that represent the public interface or the facade to that module. In this case, the getCities, saveCities, and City class are all going to be exposed on the object that was returned from the require of API. It's the responsibility of the module definer to expose the facade of this dependency, so you're exposing that API. Now this could be simply different properties on that or it could be one property that exposes a bunch or a complicated interface. It really depends on how you want to define it and there's good examples of both. Lastly, the dependency management in AngularJS is pretty simple, has the notion of objects, of services, of directives, of factory objects by name. In this case, from the module we're creating a new factory called a dataFactory, which is simply a singleton module that knows how to deal with data. It's going to expose some data and probably have some functionality like get data from server, save data to the server, etc. It's going to be an object that represents a way to get data into our system. When we need it, like in our controller, we can simply pass it in by similarly specifying the name of the dependencies we need and then expecting it in the callback. The first time that someone requires this dataFactory it's going to execute the function inside the factory method and take that object that was returned and pass it into, in this case, the controller. It's simply doing these by name, so when you're defining modules in AngularJS you're doing it by name and then the resolution of what dependencies you need are by name as well. You'll notice in the factory method there's an empty array accessor in that this dataFactory doesn't have other requirements, but it could. We could include other dependencies that the dataFactory needs and it would walk up that chain of dependencies just like AMD or CommonJS do as well. One thing to note is that it's defining it in two places. It's defining the dependencies both in the array and as a pass back to the function because we may need to minify the JavaScript we're dealing with here. The function name here, in the case of minification, is probably going to be simplified to a single variable or obfuscated to a garbage variable name. It's still going to work, but we're not going to be able to depend on the name of the function parameter, so we have the array of dependencies names to use to pass into that. We have the array of names that won't be affected by the minification and then it's passed in by ordinal to the list of function parameters. Smart Asynchrony Most JavaScript code we're going to be dealing with is going to need to deal with asynchronous calls. This is often because we're doing operations that require asynchrony, so we may need to call up to the server and get some data and wait for that data to be returned when it's possible. If we didn't use asynchronous requests we would tie up the UI while we're waiting for those requests. It's become a natural habit of dealing with JavaScript and asynchronous calls, but a lot of the code we see today in JavaScript ends up being terse and hard to read. We want a smart way of dealing with asynchrony. The problem is that using deeply nested callback stacks are really hard to maintain. We would like to rely on existing patterns that some code is using today or even develop new patterns. The two patterns we're going to talk about are promises and Async libraries. Let's talk about the problem here. He used JavaScript that was implemented for a fairly simple page and the first thing it does is goes and gets one set of data. In this case, that set of data is going to be some destination data on a server. We have to wait for it to return before determining that we need to make a second call to get some information about the user. We need both of these pieces of information. We could try to do it in parallel to get them at the same time or we might need to do them in this nested fashion where we need the result from the first to do the second, but we very quickly get into these very deep trees of callbacks, and this is what we want to avoid. The first solution is to use promises. Now the idea of promises is pretty simple and hopefully I'll show you exactly what it looks like both in using and both as a consumer of a promise and the producer of a promise, but we see promises in a number of places. I'm going to use the example here using the Q.js, which is a pretty standard promise library. AngularJS has their own promise library called Q as well. There's also a promise library inside of jQuery, so if you're already using jQuery you can actually adapt and start using promises today with existing code. The idea behind a promise is that instead of passing in a callback our calls are now going to return this object called a promise and a promise is going to allow us to define certain methods that are going to allow us to chain different callback mechanisms. In this case, we're calling some module that has a makeAsyncCall function on it, and this function returns a promise. From this promise we can use the then function on the promise to say when this comes back then run this piece of code, run this function, some anonymous function I'm going to supply. I can also have a number of these, so I might have separate operations I want to do from the AsyncCall, so I can call multiple thens, and the reason this works is the call to the then function returns that same promise, so we can chain these together. We can also check for fail by calling the .fail, so in the case of a failure it knows to chain this down and immediately call the fail. It's not going to call the then because the calls to then assumes it's going to be a successful call. We can also have code that runs regardless of whether it was successful or failed. We can do clean up code by including a finally and then we can call done to complete the promise chain if we need to. In this way we're doing the same sort of callback mechanisms, but we're giving ourselves a lot more leverage to do it in a way that's going to read more succinctly. We know that when this asynchronous call is made we're going to do this, then this, we're going to do this if it fails, we're going to do this regardless of how it works, and then we're complete. It allows us to make code that's more readable instead of this endless nesting of callbacks, which we've all seen in JavaScript. Depending on the libraries you're using you're going to be able to produce these promises when you implement the makeAsyncCall as well. It's you're responsibility to go ahead and implement promises, as well as consume promises. In the case of something like jQuery, you can actually call the $get or $ajax and use promises today. The problem is some developers aren't aware it's there and so they end up passing in callbacks and for backward compatibility jQuery continues to support those. The other pattern here for dealing with asynchrony is the Async Library. This is a library that is commonly used in NodeJS, as well as client-side code. The interesting thing about this library is it's going to allow you to do things like take these bunch of operations and do them in parallel and return back to me, or do them in series if they need to be done and then callback to me when it's all complete, and allows you to handle complex situations in a way that's a lot more readable. For example, we can call async.parellel, which is one of the methods and pass it in a function that is one of the operations we need to do. Notice that this is being passed back at callback so that it can use the old style callback mechanism. We can even have these two functions. Now, with async.parellel, we're telling the Async Library here are two functions and call them at the same time, allow us to wait for both of them. Optionally, we can have a callback function ourselves that is going to contain all the data from all of these parallel operations. In fact, in this case for the results that are passed into this final function, it's going to be an array of the results from each of those operations. We can see the error for each of the operations, as well as the results for each of those operations, so we can make smart decisions about how to deal with very complex mechanics. Now if you've ever done anything like work with the Amazon catalog API, you know that getting information, let's say on a book that you want to sell through your website, isn't a trivial operation. A single call to the API is rarely enough. It's often a chain of these calls and doing that in a way that is not only maintainable and readable, but also efficient is difficult, and the Async Library can certainly help you do that. Loose Coupling The last technique we're going to talk about in this module is the notion of loose coupling. Though we talked about dependency chains in module definitions, so we know how to define a module, we know how to supply dependencies to it, but what if we want two modules to speak to each other? Do we always need to be able to pass one as a dependency to the other and you end up with sort of the embrace where they both need reference to each other and how do you know which one to create first? Loose coupling avoids that by saying that modules can communicate without having to be a dependency of each other. The trick here is to not maintain hard links between these modules, so instead of having a reference or a dependency chain for each of these modules, you're going to want to allow them to communicate without having reference to each other. The reason this is important is when you're doing testing you don't want to have to craft up every module that you may need in order to do the test, and in this way you might want a module to send out a message, even if there's not a listener that can receive it, and vice versa. You may want to receive a message even though one is never sent. This is where loose coupling can be really useful. In this way, we avoid every module requiring a reference to every other module, which in a complex system in a dashboard system, or rich single page application, or rich web applications can be pretty common. I see a lot of code out there that ends up doing all of this meticulous wiring to make sure everybody has reference to everybody and that, at the end of the day, is going to cause you a lot more maintenance headaches because that wiring ends up being a maintenance task in itself. By avoiding loose coupling we're going to enable testing in a pretty simple way. We can do this by using messaging. We're going to do this by either using a publish and subscribe model or just using Global Events. Let's see how all this is done with a couple different libraries. The three solutions I'm going to talk about, and they certainly aren't the only solutions out there, are jQuery events, so that even in the simplest project using jQuery we can see how to do this loose coupling. We're going to see how a library called AmplifyJS can simply enable us to do this, and we're going to show it in AngularJS as well. In jQuery there's this notion of a global event object that's simply on the jQuery object and we can trigger our own events. When we think about triggering events in jQuery we may think about triggering the click event, or the form submission, or the hover event manually. You can actually do this to trigger specifically named events, so here we're going to publish an event in one module called our.event.name, and you can give it whatever name you want. It's often useful to use prefixes for your type of project so you don't run into someone else's event name that they may be using in their own project. The second parameter is a list of the context or the data associated with this trigger. If it's really _____ fire that something happened and there's no context you don't need to include any data, but if you do include it here when we then, in jQuery, use the on to wire up an event handler for our event, and notice that the names here are the same, then we can get this event handler that is going to include the event itself, and then one parameter for each of the pieces of data that are passed into the event. Now instead of passing in an array here you can pass in on an object knowing the object will simply be passed down, but it's often useful to pass it in as an array so that the callback can get it as individual parameters in the function callback. The downside of jQuery is that you have to have a jQuery object, a DOM object, in order to wire up the event handler, so we have to go out, whether it's on a document or on an individual node, and wire this up. Even though this is a global event itself it doesn't really matter what jQuery object you're listening on, it's when you find anyone that has that caller. AmplifyJS has a very simple model and is a library I tend to use quite a lot in order to handle this messaging and we can see that the publish is very similar to the jQuery. It's simply amplify.publish, a unique message name, and then context as parameters to the publish. On the other side you would subscribe to that same message, so the subscribe may be in one module, the publish may be in another module, and you can have this communication go back and forth, where the publish is always sending a message, and the subscribe is always listening to the message, and you could even have a publish and subscribe on both sides so you can have bidirectional communication. The key here is that these aren't necessarily going to be in the same modules and is going to allow you to communicate without having reference to those other modules. You're simply going to have to have a way to know that the name of the message is the same on both sides of the wire, and that may be another module that contains message names, or that wraps this whole process, that both of them actually use, a common module that they both use to communicate. Finally, Angular has a built in system for doing this, and that is on the rootScope object, which is a dependency you can use in AngularJS, there's a broadcast function, and this should look similar because it's using the same semantic, the whole idea of publish and subscribe are so simple that most of the implementations of this are the same. Here we're going to broadcast a well-known message name and then some context, and then in some other part of the system, like here in the controller, we can take that scope and say if anyone publishes this message, on much like it was in JavaScript, then call this callback so we can handle that event. Using these events, or public subscribed systems, are going to allow you to build loosely coupled modular code that can talk without having to have a reference to each other. In this case, we have two different controllers and the controllers don't need reference to each other they just need to know that one can broadcast and one can listen or they can both listen to each other's broadcasted messages. Summary Let's wrap up this module. We've seen some techniques from creating more maintainable JavaScript so that you can build large scale JavaScript projects and make sure that they're more maintainable over time. First, we saw how to avoid the Global Scope so you have to worry less about collision with other components, other modules, other libraries you're using. We've seen that using strict JavaScript can highlight these errors earlier that you may have in your code without having to wait for testing or wait for you to run JSLint over that code. We focused a lot on creating modular units of work in order to increase the stability of the project, as well as the maintainability of your project. We've seen how important dependency injection is, so we don't have to worry about that wiring up of dependencies, and we don't have to worry about the chaining of dependencies when supplying them to individual modules in our project. Hopefully, we've talked you into abandoning nested callbacks in favor of using other techniques like promises and Async Libraries in order to write asynchronous code that is more readable and more maintainable. Finally, hopefully we've talked you into using eventing and messaging to loosely couple your modules so that they can talk to each other without having hard references to each other. This has been module two of Large Scale JavaScript on the client and server. My name is Shawn Wildermuth of Wilder Minds. Thanks. Scalable JavaScript Introduction Welcome to the third module of Large Scale JavaScript on Client and Server. My name is Shawn Wildermuth of Wilder Minds. In this module we're going to be focusing on scalable JavaScript. We're going to talk about how to improve code, optimize code, and ultimately compose code. Let's get started. The Problems of Scale What do we mean when we say scalable JavaScript? In the prior module we talked about maintainable JavaScript, which is important to this story, but we also have to deal with the problem of scale. As the amount of code you're dealing with becomes larger you need to be smarter about how you're using that code. Now this JavaScript code we're talking about could be in a very complex web application. It also could be the backend of that application in something like NodeJS. The problem of how to deal with lots and lots of code in these different scenarios is something you're going to have to tackle. We've all used web applications that dealt with some of these scalability issues like slow start up, slow running JavaScript, unresponsive JavaScript or simply pages that were overwhelmed by the size of the rendering code. We want to find solutions to these problems and we're going to focus on three different ways to optimize this code in order to increase its scalability. Now no matter what you do, at some point your JavaScript code at the end of the day will have a ceiling to its scalability. There's no way to make it infinitely scalable, so you're going to want to make sure you have as much head room as possible. This head room is important because we're talking about running the same application on lower end devices like phones and tablets, as well as high end devices like full desktop browsers, and you want to allow your application to scale to those different devices well. We also don't want to scare away the users and we do this by making sure that the application is very responsive, that when they start up and they get to the page we're going to present the page to them as quickly as possible. We may include things like loading screens and hour glasses in order to tell them that we're still working in the background, but we want to give them access to at least the bare minimum facility in the webpage as quickly as possible. JavaScript tests three major scalability problems. These include Runtime performance, too much JavaScript, poorly performing JavaScript, and Runtime performance is really hampered by the difference between browsers and whether you're mobile or nonmobile. These differences really matter to the performance of the JavaScript because the underlying processor has to be able to run these and the underlying JavaScript engine has to have the ability to execute this code quickly. Not all engines are the same. Different browsers, different platforms are going to have different performance characteristics. We also have a problem with parse time. This is going to affect start up time, as well as deferred time later in your application. Every line of code in parsing is expensive. We're going to find ways to minimize that and make sure that you're delivering just the core experience to the user and nothing more. We also have the problem of download speed. SO the size of your JavaScript becomes important in these cases and so making sure that the size of your JavaScript is small is important. You can't assume that JavaScript is going to be cached in the browser in every case. There have been studies, there's a famous one by Yahoo, where they found 60% of hits to JavaScript were not coming from cache. If you download and release your code very often you'll find this to be the case as well, as new versions of your code are going out. There are some solutions we can focus on. We can improve the quality of our code to improve that Runtime performance, we can optimize our code to improve the parse performance, and we can compose our JavaScript by not including all the JavaScript in one large lump sum as the page loads, and we're going to see in this module how to do each of these. Improve JavaScript The first of these three steps to making our JavaScript scalable is going to be improving your code. This includes writing less code, using good App architecture, and dealing with the UI in a smart way. So let's talk about writing less code. The size of your code can impact performance, and we're not talking about optimizing it, we're going to talk about that in the next section, we're talking about fewer parts of your code. Parsing that code is part of this bottleneck, and the time it takes to parse your code is not in direct relation to the size of that code, and so literally writing less code or writing in smaller pieces is going to be important to making sure your JavaScript is scalable. Your code is also not necessarily JIT or object cached in all instances, so you might think once this is loaded in the browser, and especially as I'm pulling this out of cache I may be getting that benefit, but it still needs to be parsed and just-in-time compiled in many cases, especially in the newer engines that can perform JavaScript much faster it's still doing some optimizations that aren't' being cached. An important concept here is that running no code is really quick, so getting up to speed and showing the user what they have an option for and then deferring your operations until the user really needs them is a part of this story. It's going to improve that start up experience by not throwing every piece of code at the user all at once. The user can get invested in a page once they have seen and are able to interact with it fairly quickly. The user will wait for individual parts of your UI to load as they interact with them and they'll be much more patient in that sense then they will on waiting for that initial page to load, waiting for all of the little pieces to come together, and so making sure that your code is quick and tight when it comes to start up experience can really hide a lot of scalability issues. You should also be using a smart application architecture and in module two we talked a bit about this in separating your applications into modules and using facilities that are across those modules to do that. We want to introduce a new concept here, and that is layering of your application. Now you may be using a typical application framework, let's say Angular, or Backbone, or one of those to build your application. You will have other libraries that are going to be at the core of what code you've written. They may be base libraries that do authentication or know how to do data access and those sorts of things, and above those you should have a common facade, or an application object, that ends up being the cop that all the modules communicate through and so that, as you layer modules on top of it, that as the user interacts with those modules all the communication is going to go through that facade, and in that way these are going to be very testable and you're going to make sure that each of the pieces of your application are going to be very small and straightforward. In this way, when a module wants to communicate with another module, let's say the stock price widget wants to talk to the user widget, it may be simple to make that a tightly coupled relationship because one module wants to talk to another, but we will want to stress that, like we showed you in module two, that all of this communication should go through a common facade, whether that's a messaging framework or your own application object you should never couple these modules together. When we talk about modules what do we mean? Typically when we're building web applications, if we take an application like we see here, we're going to start to think about modules as being individual parts of the UI like we're showing here. This page may be made up in this case, of six modules that have different pieces of functionality, whereas the user module may want to communicate with the folder module they're going to need to go through that facade that you're going to want to use that's going to be the mediator between all these different pieces. The moment you start to tightly couple these all together you're going to find the performance of your application is truly going to suffer to where parts of your application are going to be in a wait state waiting for other parts of your application to respond or to do something with them. By going through the facade you can guarantee that no one is waiting for one part of the UI to react. You can guarantee that one part of your UI isn't waiting for some other JavaScript call to complete before becoming responsive again. In this way you're also going to want to be smarter about the way you write UI code. The idea here is to be as lazy as possible. This means avoid pre-drawing UIs that are hidden. The idea that you're going to set them to be display:hidden means that the size of your DOM is going to be larger, so wait until you need the elements in your UI to actually draw them. If you have drawn HTML and you want to hide it, a good strategy here is to actually pull that HTML out of the DOM so that it's not keeping the wait of the DOM in the browser, but keep the HTML around so you can simply reattach it to the DOM when the users want to show that again. Simply hiding it certainly will make it go away, but it means that the DOM has to consider that as it's dealing with drawing the UI and can slow down refresh rates. It's important that if you do this you're going to have to manage invalidating your own cache, so that as things may change in that HTML you're going to have to say, oh this HTML cache is no longer good, let's get rid of it, and just redraw it from scratch next time. Recreating an entire section of your UI is often cheaper than trying to do one or two element partial updates. By digging into that DOM instead of replacing an entire section with HTML you may find that updating the UI is going to be slower and so trying to do redrawing can be quicker in many instances. Another technique that's important is to do the DOM manipulation without the DOM, and what we mean by that is remove the elements you're going to be manipulating, make the edit, and then reinsert it into the DOM. As the DOM is dealing with a live view of some element, even if it's hidden, as you make manipulation to live DOM elements it's going to affect the performance and slow down the entire process. Lastly, building up arbitrary HTML and then using InnerHTML to shove it in instead of creating one DOM element after another is usually preferable. In this way you're interaction with the DOM is a single call instead of a bunch of concatenation calls and remember the first time you insert something into the DOM the DOM now has to be aware of being able to display it and deal with different changes. By doing it as an atomic call you should find that this UI code should perform much better. Optimize JavaScript The next step is to optimize your code. This is going to in particular, improve parse performance. As we've said before, you're going to want to want to make your source code as small as possible. This is going to improve the performance of the parse step into the browser. You're also going to want to automate this process, so that you have an actual build step. It's going to take your different pieces of JavaScript and produce optimized versions and you're also going to want to remove any code sections that are only used when you're debugging your application, whether this is console.log or other sorts of facilities that should be there to help you debug situations you may not want in your final code. Simply having them not run isn't good enough. We're going to want to use conditional compilation to actually remove them from the code-base. When we talk about optimization we're really talking about minifying our JavaScript, and this is the process of removing all unnecessary characters from the source code without changing its functionality. Minification is simply the ability to compress our JavaScript or other content, but we're going to focus on JavaScript, into smaller packages. Minification is a bit of a commodity. There's a bunch of different ways of doing it out there. I'm going to show you some simple ways, but as long as you're doing minification that's the important part of optimizing your JavaScript. It is not important which one you use. It can become a very political debate about which one is saving you 2 or 3 bytes in either direction. I think it's much more important in the big picture that you are doing minification, not to argue about which one you're using. In your own code you may need to plan for this minification because doing things like relying on function parameter names is really a problem because minification is going to remove those names and use really short parameter names, so anytime you're relying on the actual name of a parameter in part of your code this is likely going to break that. You should also automate this process and you can do this with Visual Studio Web Essentials, Visual Studio Web Optimization, GruntJS, there's a number of tools, we'll actually look at a few of them, but the important part is that it should be part of your process to getting from debugging on a developer machine, to staging on a test machine, to releasing it on a production machine. Let's get a sense of what happens in minification. Here is a simple little test JavaScript, and we can see that we're using a self-executing anonymous function here, and we're just handling some code on initialization of the page. A really simple piece of code, we're not doing a whole lot here, and this is what the code would look like in many cases when we actually build it. Before minification and while we're debugging we're probably going to use this version of the test.js, and when we minify it it's going to make it into this much harder to read version, so let's see this formatted and talk a little bit about what is actually happening. I've added some line breaks and some indention here so we can see it, but it's still the same code after minification. One of the things you'll notice is the parameters, functions, and variable names are all minimized. They're often going to be one or two characters long depending on how complex the document is. Within a scope it's going to maintain that it's not going to reuse any of those variable names, so it many end up having to drop down into two letter variable names if it runs out. We can see that the function name is now t instead of init, the jQuery parameter passed into the self-executing anonymous function is now n, and even the parameter passed into our init function is named t as well. It doesn't care that those are being named the same things, it just wants to make sure that this is going to execute in a very straightforward fashion. The goal here is to keep all of the same functionality, but reduce the amount of white space and noise in the application, and you should think of minification as really the build step for JavaScript. If you're a .NET person you're used to a compilation step taking something like C# or VB and creating IL from it that is actually run on a particular machine. If you're used to JavaScript the same thing happens with bytecode. This is essentially the bytecode of JavaScript. Now it's more readable than actual bytecode or intermediate language, but it's the same central purpose. Now there's a number of options out there and if you're not already minifying in your own way, many existing projects already include minification, you should consider some of these options. GruntJS is a JavaScript Task Runner and we're going to spend some time showing you some techniques there because it's an up and comer technology that crosses technology boundaries, so that if using ASP.NET or JSP or Ruby on Rails they're all using GruntJS as sort of a specific JavaScript task oriented runner. The library with GruntJS we're going to look at is actually Contrib-Uglify, which is based on the uglified JS library to do this minification. It's a fairly popular minification option that's out there. You can also use the Visual Studio, Visual Studio with Web Essentials or even ASP.NET MVC web optimization. Those are all good solutions for doing it. WebStorm and UglifyJS together work well and even Sublime in the Minify Package. The important idea here is that you're going to do minification, not which technique you use. If you're already building websites using Visual Studio, WebStorm, Sublime or other techniques, you probably already have your own minification option, but if you're not I'm going to introduce you to GruntJS. Using GruntJS GruntJS is an Open Source library that allows you to really have a concrete build step for your JavaScript. Now GruntJS has a number of uses and a bunch of plugins. We're going to cover some of them during the course and this is going to be the first place we're going to sort of introduce GruntJS, but we're going to revisit this in the testing chapter as well. Before you get started with Grunt you're going to need to know how to install it. Grunt is based on NodeJS and the Node Package Manager, NPM, so you're going to need to install NodeJS. Now this doesn't require that you use NodeJS as your backend, you could certainly do this with ASP.NET, or Ruby on Rails, or PHP or whatever you're using as your backend. NodeJS is simply a sort of platform that a lot of this stuff is built on top. It's simply just a Runtime that GruntJS uses. In fact, the JavaScript Runtime that knows how to execute JavaScript on the client really well. Once you have NodeJS installed you're going to want to install GruntJS globally or within your project. For global installation you can simply use NPM and use install -g grunt-cli. This will be the grunt command line interface. This is going to allow us to call grunt in a folder and have it attempt to execute one or more tasks for us. When you're using grunt plugins you're going to also need to install them and installing them is a little different than a typical NPM package in that you're going to want to include the package, but use the --save-dev flag, which is going to tell it to include it in the local node configuration file, but not as part of a Runtime environment. This is especially important if you're not using Node as your backend. Once you have it installed how do you actually use grunt? Grunt is just a command at the command line and you're going to say grunt and then give it a task name. Without the task name it's going to try to look for a task called default. Now grunt is going to look for, in that directory, a file called gruntfile.js or gruntfile.coffee. It supports both JavaScript or CoffeeScript in this case. What grunt will do is look at the configuration in this field and figure out what tasks to perform. What a grunt file looks like, and I'm going to just focus on the JavaScript version, is it's going to export an object to callback, that's going to pass in a grunt object, that are going to call initial config on, and this initial config is going to be these are the type of tasks that I need to do. This is going to be essentially, a configuration of all the different kinds of things this grunt file does. In our case, we're going to simply initialize it for JSHint so that we can use grunt to run JSHint over our JavaScript to make sure the JavaScript we're writing is good quality code and passes JSHint testing. Note each of the plugins has their own configuration details so you will need to look in the plugin to see what the configuration. For JSHint this is as simple as saying, here's an array of different paths to go find JavaScript for, and in this case I'm saying, look at all the JavaScript in this particular folder or I could also use folder wildcards or even entire project wildcards to look through this JavaScript. I want to only look at the JavaScript I'm writing and so this JSHint configuration makes the most sense. I'm also including any options and in this case, I'm using some multi string literals in JavaScript and I want to tell JSLint to ignore multi string continuations if it needs to. Now JSHint has a dozen or more of these options you can set, so you can determine how rigorous or how un-rigorous you want JSHint to be. The last piece is to load the plugin into our GruntJS file and simply by saying loadNpmTask and then give it the name of the plugin. In this case, that plugin is the JSHint. Most of the grunt plugins start with grunt-contrib- and then the name of the actual plugin. This would be the same name that you would use to install the plugin with NPM. Once we've defined this in a grunt file, this list of configurations, we can then simply type grunt, space, the name of the task we want to perform, and in this case, JSHint since that's the plugin we're using, is the default name of the task we're looking for. We could also specify a default task and in this case we're telling it that if we call grunt without a name of a task it's going to call all of the tasks listed under registerTask, default, and then you'll see a list of tasks. Because we have a default task now, we can simply call grunt and it will run through all of the tasks in the array defined under registerTask. We could also define more than one task if we wanted to group individual operations together, if we wanted to run a number of tasks right after each other, maybe JSHint, maybe optimization, and then maybe running some tests, all of that could be handled through grunt. We started the conversation by talking about GruntJS can help us do our optimization using UglifyJS. Let's see how that would work. The UglifyJS plugin is going to help us do this and we can configure this by simply saying grunt, initial config, and then specifying the options for uglify. By default, we can just specify a build set of files. The source is going to be everything in that public JavaScript directory and then the destination is going to be a single JavaScript file that I'm giving out here. I'm saying go collect all of the JavaScript files in that directory and then not only minify them, but concatenate them into one giant JavaScript. Now this optimization step is going to reduce the amount of round trips we need to make and just create one large JavaScript that is all optimized into one long line that's going to include all the functionality in that folder. Later, we'll see another way to use grunt to not necessarily concatenate these all into one giant file, but uglify has these different options for doing that. So minifying your code is really helpful to this whole operation. It's going to make parsing that much quicker, but you're also going to need the step of removing code that is only useful for debugging. Doing this conditional compilation is crucial for delivering only the JavaScript that the browser really needs in a release mode. You may need this extra code while you're debugging, but you certainly don't want to deliver it to the final browser whenever possible. Conditional Compilation Using minification is really helpful in making the parse steps fast and efficient, but at the end of the day there's going to be times when you're going to want to remove code that isn't useful in the production environment. You want to remove any code that is only used for debugging so that the minimum amount of code is actually delivered to the browser. This is part of that optimization that's really important. If we look at a piece of code here, this is the source code that we want to actually get rid of some elements in, and we can see this first line is simply defining an object called DEBUG so that by default, before we optimize, we're going to have this DEBUG flag enabled. In this case, we can see that if DEBUG we can do a console right log or we can do a whole if statement based on the DEBUG to do the sorts of operations that may be important in a DEBUG build of JavaScript. Once we run this through optimization to turn off everything that is using that DEBUG identifier our result is going to look more like this. It's going to simply remove all of that extra code. It's not just that the line console.log other info isn't going to be called because DEBUG is false, it's that it never has to deal with this, it never has to deal with all those lines in parsing and in executing and skipping those lines. It makes your JavaScript that much quicker and in a big complex system, in a large scale JavaScript application, the amount of debug information you have may be quite large in order to identify issues and problems while you're writing, and debugging, and fixing bugs in a large system. In this way, when you finally deliver it to production environments you don't have to worry about any of that code slipping through or affecting the performance characteristics of your system. GruntJS and UglifyJS can really help us do this. It can prune this dead code and the way we do this is to set the options in uglify to specify that the global defs are going to include a DEBUG that's set to false. In the case of the DEBUG set to false, that first line in our other code where we defined it as true if it hadn't been defined already means that it will be included as false, and then the dead_code identifier in the bottom is going to see that no one ever calls that code because DEBUG is always false, and remove that from the operation. By using GruntJS and UglifyJS together we can actually create a release build of our JavaScript libraries, not just optimize them, but supporting conditional compilation in our JavaScript. In a large scale project this can achieve a lot of efficiencies in getting rid of code that you thought would just be skipped, in this case, will actually be removed from the code-base entirely. Using JavaScript Composition Part of your job, as a developer building large scale applications, is that you need to deliver your code to the browser along with other assets and so, having the ability to efficiently deliver your code to the browser is a key component in successfully scaling your JavaScript. Now if you've dealt with small web projects before you've probably just included all the JavaScript you're going to include on the bottom of every page or inside of the layout or frame element of your project, and this works, but at the end of the day you may decide that not every page needs every library, and that not every library needs to be loaded when the page starts up. It may only need to be loaded on use. That's where the idea of composition, or composing your application, comes in. You want to construct your JavaScript as necessary instead of one giant code-base. Instead of treating your entire code-base as one giant blob of JavaScript that every page is going to use, composition can help you achieve scale by having smaller pieces of JavaScript downloaded as necessary. The first part of this is to use that same optimization strategy we saw earlier, but instead of building one large library we're going to build these sort of modular units of code. It's not always going to be creating packages based on each individual module, that's not often useful, but being able to create these packages that are more use-based. An example may be that if you have some part of the page that shows a map, not loading both the Google map implementation, as well as your code that deals with the map, until someone opens that up cause that code may be large and why ever load it on the page if no one's ever going to actually enable that part of your UI. You're going to want to use JavaScript Loaders to do much of this heavy lifting for you so that you're not having to deal with or worry about what scripts are on what pages. These JavaScript Loaders may be combined with Dependency Injection or if you're already using a framework that supports Dependency Injection, like AngularJS, you may just use the JavaScript Loaders to download code as necessary. As we saw on the optimization strategy earlier, we were building all of our code together into one giant JavaScript file that we might include on our page, and that's not a bad strategy depending on the size of your code, but when you're starting to optimize into conditionally loading JavaScript as necessary, using a grunt file that will build individual sets of JavaScript is really useful. This is another configuration for uglify where we're simply saying we have a number of files we want to build. We want you to build a main.min.js and it's going to include one or more JavaScript files, and then we're going to want a base min.js that includes this list of JavaScript files, so in this way you can build these different, separate components. I don't think in most large scale projects that you're going to have one giant JavaScript file or that you're going to have one minimized JavaScript file for every single JavaScript file that you have built. It's going to be somewhere in the middle. You're going to find packaging together, let's say all AngularJS services, together in a file makes sense, but maybe each controller into their own separate piece also makes sense. It depends on how you're building this. JavaScript Loaders To help you, JavaScript Loaders can do a lot of the backbreaking work for you. It's a way to incrementally load JavaScript into the browser as necessary and this can really help the scale and timing of large scale projects so that your users aren't spending a lot of time in that parse step or in the download step before they're actually starting to use your application. We're going to look at two solutions, but let's start with a really simple solution called LazyLoadJS. This allows us to simply lazy load JavaScript one or more files and then get a callback when that operation is complete. This assumes you've used LazyLoadJS, and you'll see a link to this in the references near the end of this module, and let's say you have a button in the UI you're using jQuery to handle the click on that button, and then at that moment when they click the button we want to do something. We want to go get some more JavaScript in order to execute. LazyLoadJS simply allows you to give a list, an array of JavaScript files, to load with URLs that can be relative or absolute URLs, and then a callback to actually do the work once they've been loaded. We can see that the amount of ceremony this is adding is really small for the amount of benefit we're getting of deferring loading some JavaScripts until they're actually needed. LazyLoadJS works really well with frameworks that already handle dependency management like AngularJS. We're going to see another solution, one that's actually a little bit more popular and used in a lot of single page applications, called RequireJS. RequireJS mixes the task of script loading, as well as dependency management. The idea behind it is that when you load the script for RequireJS you're also going to specify the bootstrap JavaScript file that you want it to load as well. When we look at the main.js we're going to tell it that we need certain elements in our application and the list of dependencies here, that destModule, is going to be the name of another JavaScript file that RequireJS will go get and then compile and then callback this function with the results of that destination module. The first module that needs that destModule object will load the script, but after that anyone else that needs that destModule will get a cached version of it from RequireJS. In the destination module itself we are defining the object that we're going to return, some object that has properties and functions and those sorts of things using a pretty comfortable module pattern. Define has its own list of dependencies as well and so this will chain dependency after dependency after dependency. If define required another module and then that other module required another, require would make sure that all those scripts were loaded and parsed before each of those callback functions were actually executed. Require.js doesn't need to load these based on the main module that we saw in the script tag. You could us it in a more simple way like we saw with LazyLoadJS. Use require, define dependencies, and then use the destModules as necessary like we saw before. RequireJS is going to still need you to use their patterns, their require method, and the define methods as a pattern of building your elements. In this way require is bringing in some structure to the way you export modules and define them, whereas something like LazyLoadJS is just there to load scripts for you. It's going to let you handle your JavaScript in your own way. All that LazyLoadJS is doing is the actual loading of those elements for you. Nothing more, nothing less. Because of that, LazyLoadJS is really tiny, whereas require has a little more heft to it, but also again, is giving you more functionality for it. It really depends on what you're using as your other frameworks, which of these techniques is going to be the most useful. Measuring Scalability There are a lot of techniques we've seen here for making your JavaScript scalable. Ultimately, it's going to be up to you as the developer to measure every moment as you build your application. Measuring the scalability of your JavaScript on different devices and different browsers with different performance characteristics with even different amounts of data is required to make sure that you're creating great, high performing, scalable JavaScript solutions. None of these on their own will solve all of your problems and finding those particular bottlenecks in the way you're writing you JavaScript code optimizing your JavaScript code, optimizing your JavaScript code, and loading your JavaScript code is something that you're going to need to focus on as a developer. We've talked about a number of different libraries out there. I want to point you to a couple of references for you to do continued reading on. The first one here, which I haven't referenced before, but I would want to point out because some of these ideas come directly from Joseph Smarr's talk is his High Performance JavaScript Slides. Go view these and they talk in a lot more detail than we could in this course about creating high performance JavaScript, especially as it relates to complex UI. Here's a link to GruntJS, to Web Essentials for Visual Studio, LazyLoadJS, as well as RequireJS. Summary To wrap up this module, we've seen that scalable JavaScript is to know that you have a scalability problem as long as your application is of a certain size. From the beginning of your project creation you're going to have more success in the long run. Some of the techniques we described are going to help, but at the end of the day the biggest one I want you to focus on is making better code, writing smarter code, not just code that gets the job done. As the amount of code in your application becomes large, the quality of your code is going to directly impact how well it performs to the user and how happy the user is. In most cases, being lazy about how you're executing your code is going to be the best approach. Deferring execution of code, in essence doing nothing, is ultimately going to give the user the best experience and they will be more patient in waiting for operations to execute later in the cycle. They are not going to be as happy waiting for a bunch of optimization stuff to happen as soon as they open the browser if that's going to delay them from getting any information on a website that would make them happy. Using minification can help the performance of parsing and ultimately downloads as well, so you should be using it period, end of story. There's no excuse for delivering un-minified JavaScript to the browser in almost any situation, especially at scale. While minification helps, you're also going to want to think about only including the JavaScript that's required to do the job, so using conditional compilation is going to help you deliver better JavaScript to the browser. Late loading any of your JavaScript that is needed for a particular task can help you defer any performance headaches in downloading and parsing of individual JavaScript libraries. Smartly caching them is one thing, but being able to defer the loading of those JavaScripts is certainly helpful. You can't always depend on cache, meaning that the JavaScript files are going to be immediately available or that a user is on a device where downloading that JavaScript file may be quick. They may be on a 2G, 3G or 4G connection. You can't assume they're not using a phone or a tablet out in the field versus being at a desktop with a 100 Mb into the wall. This has been module three of the Large Scale JavaScript on Client and Server. My name is Shawn Wildermuth of Wilder Minds. Thanks. Testable JavaScript Introduction Welcome to the fourth module of the Large Scale JavaScript on Client and Server course. My name is Shawn Wildermuth of Wilder Minds. In this module we're going to be talking about testable JavaScript, the important third leg of the three legs that support our large scale JavaScript implementations. This is going to include talking about why testing JavaScript tends to be difficult. We'll explain what unit testing is, I'll use Jasmine to write unit tests, show you how GruntJS can help us execute those tests, and finally how to automate those tests also using GruntJS. Let's get started. Why is Testing JavaScript Difficult? What exactly do I mean by testable JavaScript? In the simplest terms, I mean JavaScript that's going to enable you to find bugs in your code. Testable JavaScript is simply being able to tell whether the code you've built with your bare hands is any good, whether it's in the client or on the server. This ability to test your code means that as you go forward you can have a level of confidence in the quality of the code that you're not touching. This is not just about the quality of code that you deliver to the client or that you deliver in the first run of a production website. It is the quality of the code that you're going to have to live with for the next 6 months, 6 years or 16 years. Instead of having to go back and refactor the entire set of codebase over and over again, knowing that you have some level of quality of code you know that some lower level elements or some modules or some components you're using don't need to be touched in order to change characteristics of a system whether that's adding features or improving features. This means we don't have to throw the baby out with the bath water every time we decide to reinvent or to add new features or to deliver better experiences to the people that use our software. In many websites today you're going to see a lot of code like this. It's using jQuery, it's using event handlers, and trying to do a bunch of code just to get it to work. In many ways this is the culprit of untestable code. Now some people will attempt to make testable code by simply wrapping all of their code that looks like this into modules or inside of self-executing anonymous functions and think, well I'm not in the global scope anymore, I think I'm modularized, but I'm still writing code like this and this is a big part of what makes your code untestable. The problem is that using Ad-hoc JavaScript, especially jQuery style, DOM specific JavaScript, means it is very difficult to separate those concerns, and if you don't separate those concerns the only kind of testing you can do is through the UI. All the code within that handler can only be executed when someone fires that handler. Whether that handler is simply a click event on a button or whether it's a hover event on a menu, there is going to simply be one way to get at that code, and that is through the UI. This is the thing you want to avoid. Let's see why looking at this code makes it difficult. This is a simple Save Button handler and it's doing some different work. It's putting together a trip, storing some information, and then sending it up to the server. The problem is we start this entire operation with an anonymous function, and anonymous functions aren't bad in themselves, but they're untestable. I can't call this function. It is useful only within this handler. There is no way to break out what this function does and run tests against it. In many cases you're going to have nested anonymous functions, one anonymous function inside of another, inside of another, inside of another, meaning that it's even more difficult to get at these nuggets of code that I want to see whether they do what they're expected to do. We also have the problem of side effects. We have some global or closured element here called a trip that we're changing data about. We're setting the name of this global or closured trip variable and we're creating a side effect and so being able to test that this object somewhere outside the scope of the code we're trying to test is being changed becomes difficult. We have business rules directly in the event handlers. This is really common where you're going to make sure that certain things happen based on some rules, but if the rules are embedded inside the UI code there's no way to test them. Finally, mixing in navigation is going to make it even more difficult because as we're trying to test this we can't test for either of these operations being executed successfully because we're changing the scope of our website. We're telling it, oh, go to the homepage again, and pop-up a dialogue, and writing tests against that mix of UI code, business logic, data access code, all in this one anonymous place, makes it impossible to test. What is Unit Testing? When we think about testing web applications themselves we end up talking about three different kinds of tests typically. UI tests, unit tests, and then integration tests. The difference between these is pretty simple. UI test is when you're using tools like selenium, or other tools that are going to manipulate the UI and then see the output. You're going to dive into the DOM and see whether the expected behavior in the DOM is happening, and whether this is manually with a bank of people that are sitting in a room doing this or you're writing automated tests to do this, you're focusing on the test from the user's perspective, period, so any changes to the UI test tend to be pretty fragile. Unit tests are testing the granular pieces of your system. The small units of code that make up your entire web application. We talk about these units, they could be the pebbles that make the sidewalk, as well as the bricks that make the walls. It's all these small little units of work. You're testing small pieces that are being used by other pieces. Integration tests, on the other hand, are testing higher up in the food chain. They're testing out pieces that use a number of these smaller units and so you're seeing the expected behavior of large amounts of code at once. Doing all three kinds of tests is really helpful in a complex, large application, and you probably should be doing all three, but we're going to focus on unit tests because what I've found is that making sure that your system has unit tests tends to give you the biggest payoff and it also ensures that your code is testable. If you're doing unit tests and you're successfully doing unit tests, they should inform you that the structure and architecture of your code is valid for a large scale JavaScript application. Unit testing itself is simply, as it says here, a method by which individual units of source code together with any associated data, procedures, and operations are tested to determine if they are fit for use. When we say fit for use we're talking about pass fail. Does a part or a section or nugget of code do what I expect it to do? This is a quote by the great Rebecca Murphey where she talks about testing and why testing is so important. What she's talking about here is that if you're writing tests along with your code, now whether you do it first or after you write your code doesn't really matter to the bigger picture, but when you have tests that go along with that first iteration of your code, if your tests validate that your code works, when you come back and revisit it, whether it's an hour later, a week later, or a year later, those tests are going to validate that the refactoring you've done to improve the performance and improve the readability, the maintainability, whatever it may be, it's going to prove that they continue to work. The same test should pass when you write your second, third, fourth, and fifth version. This regression of functionality of your website is also going to help when you're building a large scale JavaScript. You're going to have lots and lots and lots of tests so that you know that intrinsically, when small changes are made, that they don't break the entire system. If the tests come back as valid, you have some amount of confidence in that system even as you're changing that system. When we talk about a unit test we're talking about an atomic test that determines expected behavior and this expected behavior, these expectations are important to the overall story. For a specific input what do we expect to come out? If we're testing string concatenation, if we input two strings we expect that the output of those two strings is going to be one string that is made up of both of those two strings. If we're doing a math component and we expect to input a 2 and a 3, our expected output is going to be a 5, and this is true no matter how complex or grand the kind of code you're running is. When you send in x your code processes x and produces y. You have to know what those expectations are and that's the kernel of what unit testing is, passing in data, expecting what that data should become in the end. Being able to have testable JavaScript is directly related to your job as a developer. I've talked before in this course about being rigorous about the way you build JavaScript. If you're putting together an amount of effort to ensure that your code is maintainable and scalable, you're going to be able to test it. In some ways this is the checksum for whether you've done the right work in maintainable and scalable code. If you're able to maintain good separation and modularization of your code, your code is going to be more testable. You're not going to run into the earlier DOM-specific or DOM-centric or event-centric code because you're going to already know about the separation of concerns, know that each piece of code, each small nugget of code inside a large system, has one and only one purpose. If you've gotten this far to get ready to test the code that you've already worked on making maintainable and scalable, you're code is going to be testable. Writing Unit Tests How do you go about actually writing your unit tests? Typically, you're going to use a library or a framework for writing your unit tests and there's a number of them out there, Jasmine, QUnit, Mocha, etc. There's a number of them out there. These are the three that I tend to rely on, and I tend to use Jasmine more than the others, but it doesn't mean that if you're using QUnit or Mocha or even another library that I don't have listed here that you're doing it wrong because at the end of the day it doesn't matter which one you're using, as long as you're testing your code. If you're finding excuses to not write unit tests for the units of work that you're building a large scale system you're going to fail. Period. In this case, I'm going to use Jasmine as an example. Jasmine is a behavior-driven development framework for testing JavaScript code. It's available at pivotal.github.io/jasmine and it is supported by most test runners, so whether you're using Visual Studio or WebStorm or a command line tool for running your tests, Jasmine is probably supported. We're going to see how it works with GruntJS in a little bit, but let's dive into what a test looks like in Jasmine. As simple as we can get it, there's going to be really three pieces of most tests. It's going to be a top level call called describe and this is where you're describing some context of tests you're going to run. This describe section is really setting up a scope for a number of tests to live in. When the describe callback is executed it is then going to look at each of the different kinds of tests that are described within it. The it function is called to describe the kind of test you're going to run, then it's going to test expectations. The expectations are some code that is inside the expect and then the kinds of things that you expect on the end. In this case, we just have a true inside the expect, and then we're calling toBeTruthy. We expect that whatever is in the parentheses inside that expectation, or inside the expect method, is going to evaluate to be something that is truthful. I mention this as truthy because true in JavaScript has a very specific meaning and so as long as it can be evaluated as true in an if clause it is truthy. In this way, if the string is empty it will be evaluated a false, if a string has contents in it, it will be evaluated as true. If an object reference exists it's going to be true, whereas if it doesn't exist it will be false, etc., so it follows sort of the JavaScript mentality of these expectations. It expects this thing and then there's a number of ways you can annotate what is returned from that to determine the actual test. So at the end of it, describing a suite of tests, each of the its is a specific spec, or test spec, and then there'll be one or more expectations within each of these specs. Most cases when I'm building these, I will have a suite per JavaScript file. Now this assumes that you're writing one file per unit of work, which is something I suggest as well, so that it will often look like this where you have services and validators and other parts of your code, and then for me, I normally have a directory full of my tests or specs, and there's usually a one-to-one relationship there. For each validator I'll have a validation spec. For each service I'll have a service spec. For each underlying low-level component I will have a spec that runs tests against it. Typically, we're going to have lots of tests per suite. Here is an abbreviated version of my tests for the tripNameValidation object. This is where I can test to make sure it exists by using toBeDefined, I can show that I expect that when I verify a trip name to include eight numeric digits that it should validate, it should be truthy, this call to verify, and then I can expect a long version, one that is too long for the validation to not.toBeTruthy, and that's how you do negative sort of tests. Now I'm not, in this course, going to cover every single type of expectation test you can have out there, you'll need to look in the docs for the specific type of test, but this is the sort of thing that you're going to do when describing tests. You can see that there's not a lot of ceremony here, there's just a little bit of trapping these things in sort of a canonical structure that makes sense and these directly relate to the output you're going to get from these tests. In addition to having many tests, you can also support nested suites. As you need additional levels of structure, a describe can be within a describe, that can be within a describe, etc. You can have structure for certain components that need lots and lots of tests. It might be useful to create structures that sort of group them together so that when you find bugs, or when you're running your tests, you can really see what tests are being run and where a particular test failure is as it relates to the structure. Jasmine also supports the idea of setup and teardown, so that beforeEach and afterEach function allows you to have some code that is executed before and after each it clause. Assuming we had more than one it here, the module object would be created and then released between each invocation, and this is often useful if you have an object that you're going to get dirty during your test and you want it to be clean again. Notice we're using the mod variable here created inside that describe scope, but then, because it's a closure, because this is just JavaScript, it's going to be available in each of those tests, and this is a pretty common pattern for creating an object instead of having to create it inside of each test over and over again. There'll be cases where you have dependent objects that you're going to need. There are JavaScript mocking libraries, and these are libraries that will allow you to create sort of fake versions of other objects, and because it's really duck-typed in JavaScript, you can mock your own dependencies as well if you have simple needs. Like here in the beforeEach I'm creating a ds, or a data service that is mocking the dependency that my new SomeModule requires, and this is a pretty common pattern as well, where I might mock up whether it's using a library or by hand, the different dependencies into other systems as well. You're going to want to deal with these dependencies in kind of a smart way. If you're already using a way to deal with dependencies, like in Angular or in RequireJS, put these in and register them, as those dependencies, as being mocked for the testing as well. So when we have an element that needs a required dependency, we can simply have a mocked version that we pass in, and this mocked version may exist outside the beforeEach or inside the beforeEach. It matters whether you want to reuse it or not in between each test. The last feature of Jasmine I want to talk about, though most of the other libraries support this as well, is the idea to spy on members of a class. We saw in the last slides that we had mocked up this data service object that we're passing into the SomeModule. It would be nice if we could tell whether that saveData function was ever called on the data service, whether the SomeModule was doing the right job, and expecting it to be passed into saveData. Of course saveData is just mocked up, so it's not going to do the actual operation of saving data in some places. We might just be ignoring the result, but we'd like to know that the SomeModule actually called it when we told it to save our new object. We can do this by using Jasmine's spyOn function. This is going to allow us to say, if this data service that we have, if anyone ever calls saveData go ahead and give it a return value that would be valid, and then we can test that expectation in saying, oh we called mod.save, now let's see whether the saveData of the data service inside the mod or module had been called. We can test to see whether it was called, how many times it was called, with what parameters it was called, we can do all these sorts of tests for really complex situations of testing. This is important because it tells you that Jasmine and the other JavaScript testing libraries are fully featured, as you're probably used to in strongly typed systems. We can create good unit tests in JavaScript, not just good enough, and this will ensure that the quality of that JavaScript is good. Any of these testing frameworks are really useful for writing unit tests, but sometimes you're going to want to understand how your particular application framework can interact with testing. Most the app frameworks include support for unit testing, though usually not the testing frameworks themselves. So for example, in Angular, Angular has the idea of mocking up objects and calls, including calls that go out to the server, so being able to mock up calls to REST endpoints and other HTTP calls is built directly into that framework. You can see my course for where I covered AngularJS testing and you can see a link below for how to get to the AngularJS testing part of my course. Let's see how we can use Grunt to execute our tests now. Running Tests with GruntJS Regardless of which testing library you're using, you're going to need a way to run your tests. Now depending on your environment, you may already have another way of doing this, but I'm going to highlight GruntJS as a way to run Jasmine tests. Grunt also supports other testing frameworks like Mocha and QUnit, but I'll show you Jasmine so you can see an example of what it probably will look like regardless of which testing library you end up using. If you haven't viewed the third module in this course, you may want to go back and revisit that. That's where I introduce you to what Grunt is and what Grunt configuration is like. I'm going to just talk about the configuration for Jasmine itself. The first thing we'll need is the contrib project for Jasmine. This will actually bring in not only the Grunt pieces for Jasmine, but also Jasmine itself, so with this line we don't need to worry about going to the Jasmine github repository and getting a copy of the source code, it's all included in this one package here. Once we have that, we can enter our Grunt file .js, we can tell it that these are the Jasmine settings we want to use. We want to specify what our code is. This is the code where we want to test. These aren't the tests, this is the code we want to test and so, you'll include any of the source files you want to include there. Notice that wildcards are supported. We also need to tell it what code needs to load before our own code. This is typically going to be vendor libraries like Angular, jQuery, Underscore, whatever libraries are necessary for your source code to actually work, and then finally, the specs is a list of the tests. So in my case, I like putting specs in their own directory, that way I can just include to go get all the specs at once and run all the tests in my system. The last piece here is the loadNpmTask function of Grunt and that simply tells it that we want to use Jasmine as part of our grunt script. Once we've made these changes to the gruntfile.js, we can just use Grunt and give it the name of Jasmine to tell it we want to run the actual tests. Let's see what this looks like in practice. Demo: GruntJS + Jasmine Let's show you very briefly how GruntJS and Jasmine work together. See here I happen to be using sublime text, though you could be using really any project, and I have that gruntfile.js project, and down here I have a set of Jasmine settings. I've got a number of sources, and notice I'm using wildcards for most of the folders I'm using to define the kinds of modular pieces of my project. I'm also including a number of libraries that my code actually requires, and then finally, the specs directory is looking at the actual tests that I'm running. If we look under public we can see that I've got tests for a number of different libraries. If I look at my tripNameValidation this is going to be one of the test suites I have that's going to test a bunch of the different kinds of tripNameValidations, so I might have one of these for each of the kind of validators or services I'm running. Once I have these, it's simply a matter of running Grunt. All I need to do is type in grunt and then the task I'm asking it to do is jasmine, which is the test suite. It's going to go out, find, and collect all the objects it's going to use the PhantomJS library to run these headless without an actual browser popping up and then we see here that it ran 12 specs in just over one tenth of a second with 0 failures. If there we're failures, we'd actually be able to see the error code and what line it occurred and those sorts of things. In this small example we only have about 12 specs, but in a larger system you might have hundreds, or even thousands, of tests as you get into a large scale situation. Now coming back here and running this every time I make my change might be a little tedious. There might be a better way. Let's see how that works. Automating Testing with GruntJS One of the important ideas here is that you want your test to be automated, so that as you make changes that the tests are run for you, but if you're used to working with a strongly type language like java or C# or VB or one of those, you might think of this as something that happens at the build step, so we want to run these even if we're not really doing a full build. We want to find out a test has failed as early as possible in the operation and this automation is going to allow us to simplify the writing of tests, so that we don't write the test, go in, run it, see if it fails, and have that sort of loop of switching tasks over and over again. We want this re-execution of tests as source code or specs are changed and I like it that this actually gives us an audible call if a failure happens. So as GruntJS runs, if there's an error, the error console tends to give me little beep so I know that a test actually failed so I don't even have to see it to watch it happen. This is also another function of GruntJS and it's using a contrib project called watch. Now watch isn't tied to Jasmine it's actually used to execute a task when files change so we can install it by just including the grunt-contrib-watch project, and then inside of initConfig it's a pretty simple idea. We're simply telling it that when any of the files here change, and notice I'm using the two star wildcard that tells it to not only look in the JS directory, but any subdirectories as well, when any of those files change, when someone saves a source file, a spec file, any of those files, to go ahead and run this or these tasks. This could be a list of tasks or could be a single task in itself. In that way, we're just going to call grunt watch and grunt will sit there in a loop and run it as soon as the tasks have been changed, and then go back into another waiting state watching those files, waiting for that change to happen. Let's see this in action. Demo: GruntJS + Watch Let's use the watch plugin into GruntJS to automate the running of these tests as we make changes. So back here at the grunt file, if we go down and look at the watch declaration, we can see that we're telling it that in any of the changes to go ahead and execute the default task. Now I could just have Jasmine here to run the tests, but in my case I'm telling it to go ahead and run JSLint, uglify, which was doing the compression, as well as running Jasmine, and so it's going to allow me to run the entire build process every time I make a change. The JSHint is an interesting one in that it's going to find JavaScript errors that maybe my test wouldn't even find. If we run grunt watch, it's going to say I'm running it, and now I'm just waiting for something to change. The first execution doesn't actually run the task it waits for a change to happen. If I open up this validation and I simply added, let's say a space, and save it so a change is happening, we'll see that over here it made a change and ran the entire set of operations. It ran JSHint, it ran uglify, and then ran our Jasmine tests with the 12 specs completing pretty quickly. If we open up one of the tests, and let's change this test to being not defined, let's save it, and it went ahead and built, but we can see that we actually got a failure in jasmine because they change that behavior, and what it does is it puts all these periods here and then an x in the test that failed. TripNameValidation failed, expected it to be defined, and it's going to show you that it failed because it expected the verify function to not be defined, which is in fact what we did here. We said expect not to be defined. Let's save it again, it's running, and it all completed. So we can follow this operation over and over again. We're working on our code and we're waiting for a failure to happen if one is going to happen. Now some people will have this on a separate monitor, some people will simply listen, like I do, for the ding. In either case, we can automate this process so we can spend all our time on the code and not worry too much about waiting for these things to happen. In fact, because of the way that Grunt works with the Jasmine test runner, we can continue to write code every time we press CTRL S to save a file and just assume that this is running in the background. It's not going to lock any of the files, it's not going to keep us from making those changes, we can just go about our day and do the work we should be doing. Hopefully this, and the other things we've shown you, are going to simplify the ability to write these unit tests and execute them in large scale JavaScript projects. Let's wrap up this module. Summary In this module we focused on testable JavaScript. Hopefully you've seen that event-driven, nested JavaScript is simply hard to test, that mixing too many of these concerns are a big problem to writing tests, whether they be unit testing or integration tests. You could still test your applications using UI testing, but in large scale JavaScript the amount of UI testing required could just be overwhelming and not test all pieces of the code you're actually dealing with. If you're focusing on making your JavaScript maintainable and scalable, you should find that the resulting JavaScript should be relatively easy to test, and in many ways this third leg of the stool of our scalable JavaScript is there to validate that you are indeed making maintainable and scalable JavaScript, that you're not just hiding the old style JavaScript DOM-centric, event-centric JavaScript inside of modules. In either case, you're JavaScript is still not going to scale well even if you're using ideas like modularity and separation of concerns. If you're really still mixing those concerns inside of your modules you're in the same amount of trouble as you would be if you hadn't done that. Ultimately, writing unit tests is really important. I wouldn't get too bogged down in the problem of dogma of worrying about whether you're using BDD or TDD or one of the other processes out there that are pushing you towards a way to write software. Now if you are comfortable using test driven development, behavior driven development or any of these star DD processes out there, use them. They're very useful, but don't get too caught up in the dogma of one of these processes being the one process that works for every organization and every developer. The important part is that you're doing unit testing, not necessarily the method you're using to write unit tests. Hopefully you've also seen that using Jasmine and GruntJS is a great combination for writing your tests and executing these tests. Lastly, hopefully we've talked you into the automated, continuous testing idea of as you make changes to the source code that your tests are simply going to run, and run every time you make one of those changes, so that you can find out about errors or problems or failed tests as soon as they occur instead of making tons and tons of changes, and then later on finding out that a thousand of your tests have failed. This has been the fourth module of large scale JavaScript for Client and Server. My name is Shawn Wildermuth of Wilder Minds. Thanks. Large Scale JavaScript in NodeJS Introduction Welcome to the fifth module of the Large Scale JavaScript and Client and Server course. My name is Shawn Wildermuth of Wilder Minds. In this module we're going to focus on scalable JavaScript in NodeJS. We'll start out with a brief introduction of what is NodeJS in case you're not familiar with it. We'll show you how JavaScript on the server is different than the JavaScript you're used to writing on the client. We'll dig into writing maintainable JavaScript in NodeJS, scalable JavaScript in NodeJS, and testable JavaScript in NodeJS. Let's get started. What is NodeJS? In this module we're going to focus on server-side JavaScript and today server-side JavaScript means NodeJS or Node JavaScript. Node is a simple server technology that allows you to write JavaScript on the server and write it in a scalable way. What is interesting about the technology is there is some promise for being able to share code between client and server and not having to switch between a server language and a client language. There's some great benefits to not having that context, which in your mind is a developer, and so it's getting a lot of interest right now, but what exactly is NodeJS? The marketing slide for NodeJS talks about it being a platform for executing JavaScript based on non-blocking calls to things like disk systems, memory systems, databases, and that it uses an event driven model that enables fast, scalable network applications. While Node in many ways is a Runtime for JavaScript, and we can see a lot of tools out there much like the GruntJS tool we saw earlier, it is actually using Node to execute itself, but at the end of the day what most people are using Node for is to actually build scalable web applications. The simple explanation of Node is that you're going to use JavaScript to build web applications using server-side frameworks. There are some frameworks that cross both ends, but those are usually utility frameworks, like Underscore is a good example. The nature of writing JavaScript for NodeJS is pretty similar, but the frameworks you're going to use are going to be different, so certainly when you're writing your web applications you're not going to have that cognitive shift or that context switch in your brain between some server-side language, like PHP or C# or Java, and what you're going to use on the client side, which is typically JavaScript in HTML. There are some challenges in NodeJS, but how does NodeJS handle large scale applications? The reality is that much of what you've learned about JavaScript is applicable to the server-side as well. NodeJS supports the idea of modular JavaScript out of the box. It's a concept that is easier to work with modular JavaScript than it is just large script base JavaScript. It's built in a way that encourages you to write modular JavaScript. It encourages a non-blocking or asynchronous model for your code. It's going to encourage you to use callbacks and promises and Async libraries to make sure your code isn't stealing that extra thunder from the server. We'll talk in a minute about how the servers are different. It also treats networking and WebSockets as first class citizens, so when you want to do rich networking code, not just the nature of web applications, which typically are request and response, but if you want to actually do networking code with things like WebSockets you can do that out of the box and they're treated as first class citizens. NodeJS itself is a low-level, fast platform. It's not written in JavaScript, but JavaScript is the Runtime environment and that's an important distinction to make in your brain. A lot of people look at NodeJS that haven't touched it and think it has to be slow. The code that it's running is JavaScript and JavaScript of course, is a scripting language that we've been programmed to think of that as slower code. The code that executes that JavaScript on the server for you though is really tight networking code. It's all very very fast regardless of what platform you're on and it's also across platforms so you can run it on Windows, on Linux, on Macs. Most of the cloud providers have Node support as well, so that you don't have to deal with the nuances of getting it installed on systems. Let's look at the way that Node is different from traditional web servers. Traditional web servers are like an ATM. If you've ever driven up or walked up to an ATM, especially for some reason about 11 o'clock at night, there always seems to be a line of people that need to make a transaction, whether it's make a withdrawal or make a deposit or whatever they're going to do, and they're lined up because the nature of ATMs is the nature of the way many traditional web servers work today, and that is I come in and I'm going to consume the ATM for the lifetime of my transaction. I'm going to put my card in, punch my little numbers, wait for the magic to happen, put in my deposit or wait for the money to come out, whatever it is. Now we can build a bunch of ATMs to serve more than one person at a time, but once those are full we tend to queue up or have a line in front of each ATM. This means that most traditional web servers today are a bit synchronous in nature. When you come in they want you to consume some resource, normally a thread that services you until you're done In Node this is a bit different and it's much more like a coffee shop. If you go to a Starbucks anywhere in the world you'll be familiar with this model, and that is you make your request, you may even pay for your coffee, but you don't get your coffee. You go over to this line and wait for that coffee to be made, wait for that request to be fulfilled. In the middle of requests being fulfilled the coffee shop is free to service more requests. Now these requests may be fulfilled rather quickly. If the person in front of me just wants a water or just wants a biscotti cookie, they can hand them that, take their transaction, and they leave. They don't get into that queue for a longer-lived or a longer operation and this is how Node handles it. It allows us to write code that is going to not consume server resources when we're just in a wait state. That's what they talk about non-blocking I/O and event callback, so that much like in a coffee shop, when the order for a person in green is ready, someone calls out and calls back to them and go, here is your coffee, come get it, and that's essentially a model for the way NodeJS does this. Let's look at this in a more traditional UML diagram to see what we're talking about here. In their traditional web server the client comes in and makes a request and then the server consumes some resource, normally a dedicated thread, though this is different on different platforms, to service that request. If we have operations, especially synchronous operations, which are not uncommon on the server, we're going to consume this thread the entire time we're doing all that work and wait for the response back. In Node it is similar except that when we make a request the server does some work asynchronously and that while it's doing that work the thread is free to service other requests. If the request is quick your thread may be used to service two, three or four requests while it's waiting for your response to come back. If you've made a larger request you're thread isn't going to be consumed the entire time we're waiting for the database to return some data, or another web server or web service out on the internet to return us data. We're not consuming that server resource when we're simply in a wait state. Once that operation completes we then consume a thread again, maybe the one we had, maybe a new one, and then return that response. We're using those server resources in a very small slice. It requires that we think about asynchronicity when we write this server-side JavaScript, but we're already used to some of the same behavior on the client because on the client if you try to do work synchronously you tend to tie up the UI thread and so the browser gets locked up so that no updates, or no interaction with the page can happen, and so most JavaScript developers are already used to this model, we're just now applying it to a server-side context. Is JavaScript in NodeJS Different? One of the lingering questions here, is the JavaScript I'm used to writing on the client any different inside of NodeJS? If you've learned from the first four modules in this course how to write good JavaScript, you're going to find the application of that is going to apply to the server-side JavaScript as well. Inside Node there is an appreciable difference in the way you write JavaScript from the client-side code you're used to writing and that is all the JavaScript is executed under Google's V8 JavaScript Engine. This is a really important concept in that, we have a predictable execution model. One of the pain points in writing client-side JavaScript is we don't know where it's going to be executed. We don't know which browser, how old that browser is, what kind of device. Are we talking about an older Android operating system with an old browser? Are we talking about an IE6 machine on a Windows XP machine? Are we talking about, you know, a brand new Chrome installation with really fast JavaScript execution? We don't know and so we do some defensive coding in that case. Inside Node our JavaScript is predictable. We know exactly the environment under which it's running and that means that we can plan for that. We don't have to be as defensive about our coding. As long as you're using NodeJS 0.5 or 1 or later, which is a fairly old version at this point, 0.10 is the new sort of standard for Node and what most people are using or newer versions, and this means that Ecma 5 Script is supported in Node, so we can take advantage of the newer spec to use different pieces of the language that we may not feel comfortable using on the client where we may be running into older browsers. V8 supports just-in-time compilation, as well as an optimizing compiler, so the execution of the JavaScript is in a fairly optimistic environment. We can expect our JavaScript to not only parse and execute well, but also be optimized without us having to do all the work of hand optimization that we may need to do on the client. The big story here is that lots of JavaScript problems go away because you have complete control over your environment. You know what version of NodeJS, therefore you know what version of the V8 JavaScript Runtime you're running against, and you can make those assumptions. This is much like any server code. You know the environment under which it's running. If you're a .NET guy you know what version of the .NET framework you're running under. If you're in Java, the same story, you know what version of Java you're running under. Even PHP this is the same story. So one of the challenges in Node when you're writing especially large applications, is to think about using those newer features that may get you cleaner code. As Node goes forward it's going to be able to adopt to the newer language features faster than client-side code ever could and you can certainly take advantage of those changes. Is NodeJS Magically Scalable? If you've spent any time in user groups or conferences, you've probably heard a lot of what I'm talking about about Node. There's a lot of excitement about it, especially for frontend developers that want to get into backend server development, and there's sort of a problem here, as it is with many young application environments, and that is there is a lot of excitement and not much criticism. The community, when you hear talks given, are all about, well NodeJS is much more scalable than x, y or z technology and we should focus on that, but why is it considered more scalable? We've talked about NodeJS encouraging less blocking, but you can accomplish the same thing in other backend environments. Java and .NET certainly support the idea of non-blocking I/O, but you have to understand multithreading to accomplish it. It's not the experience out of the box. While many backends like Java and .NET tend to be synchronous code, you can write it as asynchrony, it's just not the default behavior. The default behavior in Node is to work with non-blocking I/O and to have event callbacks when operations are complete. It's the natural way of writing code, so it's sometimes easier to get scalable code running on the server is by using NodeJS because the model is different. If you are already invested in other backends you can get some of the same behavior as well, but you have to start thinking in that way. Do we want to consume those resources the entire time we're trying to process a request. NodeJS also is considered more scalable because it has this notion of scale-out, but each Node process itself is single threaded. This means a single thread, or a single execution of node, can handle sometimes many more requests than their other counterparts, but again, some of that has to do with the programming model, not the efficiency of Node itself, but Node on its own is single threaded, so it expects that if you need to scale it out you're going to need to scale it out vertically and horizontally. We'll talk a bit about that in a few minutes. As we've said, the key to the scalability tends to be that a thread is able to answer other client requests while you're waiting on other operations that would normally block, like I/O. Many of these arguments about the scalability of NodeJS isn't about the JavaScript. This means when you want to scale Node we're going to do everything we can to understand how to write scalable JavaScript, that's the name of the course, that's what we're focused on, but I don't want you to get too wrapped up in what you have to do to make your JavaScript scalable if you aren't dealing with the problems of scalability for the rest of your server components. When you're scaling Node you have to think about how to scale your Datastore. Are you using Memcaches or other techniques for scaling out the data storage that you're using in your application? You're also going to need to deal with scaling out, having multiple instances of Node run in a single box or even how to do that within something like a web garden or a web Farm. You also have to think about geographic scaling. How do you get into other markets so everyone isn't traveling the entire globe to get to your server somewhere in the data center 4 blocks from your business? You're also going to need to think about load balancing. How do you keep the instances and even the individual boxes from being overwhelmed by getting stuck with certain types of requests? None of this is new to other technologies and people in Node are thinking about this here as well. Some of these are even outside the scope of solving them with Node. When you think about scalability in NodeJS don't get completely hung up in that if you write all your JavaScript exactly perfect that your solution's going to be instantly scalable, that's not really what we're talking about. The key idea here is to know that Node doesn't do any of these out of the box. There are solutions that layer on top of it that do some of it. There are solutions that have nothing to do with Node that do some of it, and some of it are being homegrown in order to handle scalability issues. In Node scaling vertically works. This means that if we want to give NodeJS more CPU, more GPU, more memory, more disk, for the most part Node can handle it. There are some caveats in that a single instance of NodeJS is single threaded, but you can run multiple instances on a single box that will give you some sense of scale out to consume as many of the resources on a box to serve up your networking or web needs as possible. So far, there's no support for scaling it out into GPUs as well. There are some projects out there that are attempting to do this, so I suspect we'll actually have scale out into GPUs as well, but a lot of the work you're going to end up doing in Node is going to be I/O or file or memory bounds, so going to the GPU as a way to augment the CPU, to me, isn't super critical for many applications. If you're using fast memory, more disk, and more bandwidth on your network, Node tends to just work in those situations. Where it gets harder is in horizontal scaling. As we take your Node application and want to move it to multiple servers, to introduce a garden or a Farm mentality, we need to think about it because there's no standard built-in support. We can build it because the networking code in Node is pretty sophisticated, but it's not a trivial effort. Typically, I suggest that either you wait for Node to find libraries to do this or just let partners do it. If you're comfortable with your data and your web applications in the cloud, like many many of us are becoming, just trust the big guys out there, AWS, Azure, Heroku, whomever that you trust. Let the cloud partners there deal with the scale, and most of them have automatic scaling solutions, both on a per box basis, as well as a multiple cores. For the rest of this module we're going to really focus on making JavaScript work really well in Node because these larger issues of how to scale Node itself are probably deserving of their own course. This is a lot to think about and a lot of this is still maturing. We're still in a 0.10 version of Node and so in many ways it's still a pretty immature technology. It's doing some great things out there, but don't think for a second that it has a great ecosystem around it yet. It has some really good ideas out there and some great libraries, but we're not quite there where we can just drop it in everywhere and it's a solution for every large scale system we're going to be building. Large Scale JavaScript in NodeJS What about large scale applications in NodeJS? How are we going to think about making sure that the code itself is super-efficient and highly scalable? How are we going to deal with lots and lots of code in NodeJS? Unsurprisingly, it's a way that you already know. As we talked about earlier in this course, we talked about JavaScript needing to be maintainable, scalable, and testable, so that it's modularized and separated, that it's compose-able and loosely coupled, and finally, to encapsulate the functionality so it can be tested, and to use test facades in order to allow your code to be unit tested. This is no different in NodeJS. The nature of the kind of code we're writing can be dramatically different, but these three pillars of what I think is important to large scale JavaScript isn't any different. You're going to take these same ideas and at the end of the day you're going to have to be a rigorous developer if you're building large systems, and not just you, everyone on your team needs to be rigorous. The leads for each of the teams in your system need to be rigorous and then the managers over them need to be rigorous about doing things like making sure that there's code reviews and unit tests are being run to cover most of your code so that you can build these large systems over time and that they continue to be maintainable, scalable, testable. Maintainable JavaScript in NodeJS For JavaScript inside of Node the maintainability question is very much like we saw in the second module. Using modularity and dependency management are key here to making sure that your code, as it grows large, is easily editable, consumed, and tested. For dependency management, Node has a built-in idea to use the CommonJS spec and has this idea of require and exports that comes directly from CommonJS. While you can plug in other models like RequireJS or build your own, it's usually recommended that you just go ahead and learn to love CommonJS, how it works in Node, and embrace it. The way it works is pretty simple. Inside NodeJS code you're going to use the global function called require to go look for code that you need inside your application. In this case, we need a file called data. The name of the package is going to be relative to the root of the project and is going to be either a package name, if you have registered it in that way, a directory name, where it's going to look for a specific JavaScript file inside of it or a file name, and in this case at the root of our project we have data.js. That data.js contains a modular piece that's going to give us some data access inside of our application. Once we do this require for data we're able to use this object immediately to call methods, properties, wire up events, whatever we want to do with it. It is an object that contains whatever was required. On the other side of the wire there's a keyword called export. So anytime require is called to some piece of code, that piece of code has access to an object called exports that we simply add members to the object that we're exposing. As we can see here, when we require the data the object we get back has a foo object, so we're able to access it here through data.foo because we said export it. This simple expansion of whatever the API is loves to add one or multiple properties or functions to the module that is being returned from the require. A common pattern I see a lot in NodeJS that I quite like is to sort of mix these. We can use a self-executing anonymous function to create sort of a scope, and enabling strict JavaScript as you might like, and passing into that self-executing anonymous function I'm going to pass the exports object. I'm using this so I can alias what I'm really returning so that the code, as we're looking at it, is going to look like the object that we're describing that is going to be returned under data. This second version of DataJS is identical to the first version. It's a little bit more verbose, but I think it's a lot easier to read. It also allows us to introduce the idea of strict JavaScript, so that our code is better, much like we saw in module two. Scalable JavaScript in NodeJS Next we should think about scalable JavaScript in NodeJS. The three things we're going to focus on are asynchronous code in our JavaScript, that we're going to use as little code as possible, and then finally, some optimization tricks for using the V8 execution engine. They're very typically, in the NodeJS built-in libraries that you may be using like this fs, which is the file system library, they're going to use simple callbacks. These are still the old style, I'm going to pass in a function and you're going to return it to me. In addition, you can also use promises by simply including the q library. In Node it uses something called the Node Package Manager to handle adding new packages, or pieces of code or libraries, to your node project, and the package that's going to enable promises is called q. This is one that's also available on the client. Here we would require that library, and because it's called q it's going to return an object that has that API for our promise, and this is going to look very much like code we wrote earlier when we looked at scalable JavaScript on the client. We're going to use q.defer to create that deferred object that we saw when we looked at the client-side promise pattern. We're going to return that promise from the deferred operation so that people can handle the standard promise pattern of using then or finally or catch, depending on the library, and then as the operation succeeds or fails we can just use reject and resolve, again, just like we did in the client code. Another library that's pretty common in Node is the Async library, and this doesn't replace promises, this gives you another opportunity for thinking about your code being asynchronous, and it is a package called Async, so you would just simply include it by calling npm install Async, and it supports the idea of doing parallel execution of set operations. In this case, I'm showing the Async library going through the each collection. This would be just like Underscores each, the for each inside of EcmaScript 5 or the jQuery each function. This is going to create a simple loop that's going to walk through them. The big difference is that the function here, that we're passing after the collection, is going to be executed in parallel. So instead of waiting for each of these to fire and then return back, we're going to execute each of these, perhaps on different threads, and so that we're going to get greater parallelism in our code using this Async library. When this has been executed for all the items in the collection we will then get a callback that includes any errors and the results of the entire operation. This is an important idea for server-side code, in that you want to do your best job in making operations handle multiple CPUs. In this case, it's going to run enough of these at one time that it will saturate the CPU in order to finish this operation as quickly as possible. Async can also be used to execute several calls in series. This is going to call them serially one after another, so we're saying we want several operations to be done, but they need to be done in this very specific order. It may be that I need to go to one website and do something, and I can't do the second operation until the first is done. Typically, we've done this with nested callbacks, but that tends to get pretty ugly. This is going to allow us to just pass in an array of these functions to be executed and then finally, to get a callback at the end with all of the results instead of having to manage all that code yourself. Much like the each function, it also has the ability to do the same thing in parallel. In this case, we're going to pass it a number of functions and it's going to go execute as many of them as it can at once. So let's say in the case of you were building a server-side library that knew how to go and query a bunch of different services for let's say, the names of albums. This would allow you to execute all of them in just sit and wait states waiting for whomever is done first, and then once all of them have returned, either with errors or successes, our final callback on the bottom here would get executed, and we could do whatever we wanted to do with all of those results. This type of functionality can help in the client, but on the server this is crucial to gain the kind of scalability you really want. We've hinted at this Node Package Manager before. One of the keys of creating great NodeJS code is that the Node Package Manager is a little too easy to use, and so while the Node Package Manager, or NPM, makes it easy to use, it's going to be your job to go find unwanted libraries. So if you've installed things into your project that you no longer need, or maybe were experiments to see if they would solve a particular solution, or maybe you abandoned them because they didn't work well in conjunction with other libraries you had, we'll remove as many of these as possible because the fewer libraries you need means your code will be more efficient. This doesn't mean never use a third party library. often times these libraries have more people working on them and working with them, so that they tend to become better code quicker. You just rewriting your own networking libraries or Async libraries or any of those is unlikely going to outperform some of the common libraries that are out there that you get with Node Package Manager. The real key here is to make sure that you're pruning packages you aren't actually using because part of the startup mechanism of Node is to look at all the packages that are required by the project and that can certainly slowdown startup and sometimes execution speed as well. One of the things we talked about in scalability for the client-side code was this idea of minification and packaging. The clear consensus out there is that minification of JavaScript, and certainly concatenation in packages, on the server doesn't help nearly as much as the time consumed in doing this, so almost no one does this. Typically, minification is for saving time in downloading, and because all the code isn't really being downloaded, it's being loaded into memory directly from disk, the parse operation will barely be any faster, and so going through that extra operation of minifying the code, if it makes you feel better that you're saving 1 or 2 nanoseconds here or there, feel free, but it is usually not worth the effort. Node can use minified versions of your code, but in the big picture it's usually not necessary or efficient to worry about it. Performance Tips for Google's V8 Since Node uses Google's V8 engine for execution of its JavaScript, you can make some specific tuning to improve the performance of your JavaScript end Node. Let's talk about a few of the big picture items here. One of the first ideas you're going to want to think about is, what functions you're using that are polymorphic. The problem with polymorphic functions is that they can negate the benefit of caching of _____ gited. You should typically avoid it if you have a lot of different types of inputs. By creating separate functions that handle the different types of polymorphism, the engine itself can go ahead and make caches that are specific to the different versions of types you're going to pass in. When I say polymorphic I really mean, if you have a function that takes two parameters and the first time you send in strings, the second time you send in integers, the third time you send in a string and an object, you're going to get different caches of that same function over and over again, inside of the V8 engine, and every time you pass in a new set of types into that function you're going to get a new copy of that in the function cache. There are times when using polymorphism is perfectly appropriate and you might want to take the extra hidden performance for that, but typically you're going to want to be smart about passing in the same types. This is especially true if you're assuming that the types are going to be converted for you, so if you're sending in strings and then you're doing math against them so that you are getting the automatic conversion to integers, this is going to defeat that internal caching. Go ahead and convert them to numbers before passing them in. The next thing to think about are arrays. Most developers that have worked in other languages might assume, incorrectly, that V8 prefers that you pre-allocate your array, so if you have an array of 12 or 20 or 100 or 1000 items, it may be easier to pre-allocate it, and in fact, the way that arrays work best in V8 is if they grow as necessary. Going ahead and using the push method to increase the size of this array is actually going to be more beneficial to you. The reason for this is arrays themselves are not really arrays inside the JavaScript language, but they're more like a link list. The last item you should pay attention to is the fact that there is an optimizing compiler in V8. This optimizing compiler has inlining at the function level. This function level means that if you run the same small piece of code over and over again, the V8 engine can't optimize it because it's not in a function. So if taking any small piece of code that you're running many many times and wrapping it in a function instead, you're going to find that V8 is going to be able to optimize that small piece of code much more efficiently instead of letting it be just one part of a much larger operation. If you have a tight loop that is running over an operation many many times, breaking that out into its own function is not only going to aid in the maintainability of your code, but also it's going to mean that the V8 engine can optimize it when necessary. Testable JavaScript in NodeJS Finally, you're going to want to unit test your JavaScript code in NodeJS. Now what's interesting, is the same client-side JavaScript frameworks you're probably using for testing are also supported on the server. Whether you're using Mocha, QUnit or Jasmine, the frameworks themselves, the way you write tests is going to be very similar. You're likely going to need a different runner for the server-side and the client-side code. You're not going to intermingle them in one giant set of unit tests, they'll be two separate unit tests, but you can run them one after another in order to get that full coverage you're looking for. In our case, let's take a look at what happens with Jasmine. This is using Jasmine to test server code, so we're doing the same sort of writing Jasmine tests that we did on the client side, but we're testing objects that we've defined as modules inside of NodeJS. We're still defining suites of tests, we're still defining individual specs, and we're still writing expectations. Notice we're using the require function here to pull in the different required elements for our testing, but ultimately the kind of unit tests you're going to be writing are going to be very similar to client-side code. You're just going to be testing the server-side code like you can see in the slide. If you're using GruntJS like we talked about in module four, you're not going to be able to use the grunt-contrib-jasmine project to test your server-side code because there's some different requirements there. There's a different grunt plugin called grunt-jasmine-node instead. You can still batch all of these into a single task in order to test both client and server together, but you're going to have to include both plugins to GruntJS for that to work. The way this works is, you can simply install it using the npm install and grunt-jasmine-node as the name of the plugin to make sure that it's included in your project, but it's not going to be part of the node installation itself. The configuration here is pretty simple. You're going to specify the projectRoot in order to tell it where your node code is. I typically put it in an SRC folder, but you can put it wherever you want. In the task we'll go ahead and look through it, assuming you have matchall set to true, to find all the projects that are using the Jasmine library. It's looking for defines to package those up as all the different suites of tests to run. We can see there's some additional configuration here, use RequireJS, which we're not going to do, forceExit when it's done, and then some JUnit specific options. You can see the docs for how these work, but this is the basic idea. It's going to be a different set of configuration for your jasmine-node plugin then you had for the grunt-contrib-jasmine. Then typically, I will register a task that will run all the tests, and notice I'm going to include as the type of tasks to run, both the jasmine, which is the client-side tests, and the jasmine_node for the server-side tests, so that with one execution I can have it run through all the tests. I could even use watch, and setup a specific watch that's going to run both sets of tests, and that's typically what I do in these scenarios. Summary We've seen that NodeJS is a good solution for server-side development if you already are using JavaScript and have JavaScript skills. This doesn't mean you should abandon all other frameworks if you already have an investment in them in either skills or actual code, but if you're creating new projects and you want to see a different way of writing web projects, NodeJS is a pretty good place to start. Regardless of what you hear from the community, NodeJS isn't magical in the way it can handle large scale projects. It's no different from any other server-side technology in that you're still going to need bright developers doing good work in order to make sure that large scale projects can succeed. Ultimately, the JavaScript you're executing still needs to be maintainable, scalable, and testable. Using the modularization and CommonJS techniques are key to make sure you're JavaScript is going to work on the server, but if you've learned these things on the client, you're going to be able to apply them to the server much more instinctually. You won't have to work at making them modular. Node tends to encourage that sort of behavior a little more than client-side code does. A difference here is that because you have a fixed kind of Runtime for your server-side JavaScript, you're going to want to understand the Runtime environment and optimize for things like the V8 engine. Ultimately, you're still going to need to test that code, even if it's just on the server. Just because it's server code and not client code, isn't an excuse not to be rigorous in the way you develop your applications. This has been the Large Scale JavaScript for Client and Server Course. My name is Shawn Wildermuth of Wilder Minds. Thanks for joining me. Course author Shawn Wildermuth Shawn Wildermuth has been tinkering with computers and software since he got a Vic-20 back in the early '80s. As a Microsoft MVP since 2002, he's also involved with Microsoft as an ASP.NET... Course info LevelIntermediate Rating (412) My rating Duration2h 49m Released24 Jan 2014 Share course