What do you want to learn? Leverged jhuang@tampa.cgsinc.com Skip to main content Pluralsight uses cookies.Learn more about your privacy D3.js Data Visualization Fundamentals by Ben Sullins Learn how to bend data to your will with this powerful data visualization library in D3.js version 3. Start CourseBookmarkAdd to Channel Table of contents Description Transcript Exercise files Discussion Learning Check Recommended The Basics Introduction Hi, this is Ben Sullins and welcome to D3.js Data Visualization Fundamentals. D3 is a JavaScript library that is exploding on the web right now as the most powerful and stunning way to build data visualizations, and in this course, I want to start by talking about the basics of D3. We'll get into some charting, we'll look at how to work with our data and that's to say work with our data using JavaScript. We'll look at how to enhance our visualization once we have some basic stuff and we have some data to power it, then we'll get into mapping and all the cool things you can do with maps, and lastly, we'll pull it all together taking everything that we have learned from this course to create stunning visualizations. First, in this module, we're going to talk about D3 origins and where did come from and what is it. We'll look at SVG objects, what are they and how do they relate to D3. We'll take a look at actually drawing some shapes using SVG and D3, just the basics of how to actually put something on an HTML. Then we'll take a look at the principles behind how that works. Lastly, we'll bring in some data and we'll actually draw some stuff. So it's going to be very hands on and very foundational in this first module. Let's get started. D3 Origins D3js or just D3 as it's most commonly referred to stands for Data Driven Documents. D3 is a JavaScript library that works well with HTML5 and CSS3 standards to take data and visualize it all using the native language of the web. Let's take a look at where D3 comes from. Originally, when, the web started in the 90's, we had very basic things, just text on a page, images, nothing that was very interactive or visually stunning. As the web evolved, so did the technologies powering it. Things like Flash and Java we're developed and released and what that required was a heavy client side install to do all these really rich things on the web that most of the native language of the web just could not do. Things like animations and drawing. Now a lot of this was developed outside of the web in a different platform and then published to the web as a way that your browser, along with this heavy client side software could render it in a way that was much richer and deeper than the native language of the web at the time. As the web evolved though, it started to catch up and in 2005 some folks out of Stanford came up with something called Protobiz. Protobiz was another JavaScript library that basically was D3's origins and in that it was a way to take data and then visualize it on the web all using the native language of the web, you didn't have to have any client side heavy tools to install. Now there are a lot of other different ones, but this one seemed to really given the most flexibility and control over how to visualize your data and as such it actually won many different awards. Later on, those guys left Sanford and they created what is now known as Data Driven Documents or just D3 and that's where we are today, since 2011, D3 has been exploding and the growth of this has been tremendous , as well as many additional libraries having been built on top of D3. Now let's take a look at one of the foundational elements of D3 and that is an SVG object. SVG Objects SVG objects are what we refer to when we're talking about scalable vector graphics. SVG is an XML based vector image format, for creating graphics on the web and this was a standard that was adopted by the World Wide Web _____ the WC3 back in 99 and SVG came out with HTML5 as a way to draw shapes on the web all from HTML without actually having to take an image and then just have to display that on the page, but actually render the images using the native language of the web. What happens is when we have a typical image like a jpeg, a jif, of a png, those are called Raster images and what I often happens is their designed for a specific resolution. So when they're presented at that resolution, they look very crisp and clear and but as you try to scale them, or if you zoom in, or let's say you have a different resolution of your browser on a mobile device let's say, that image can become distorted and this is what SVG aims to solve, it aims to make a and image that isn't just plopped on a page, but one that is actually rendered with the browser and the HTML of the rest of the page. So with that, we're actually able to do things in a much clearly crisper way, so if we're looking at a small version of image or the large, it'll scale appropriately and it won't cause that distortion or that pixilation that you see and get with Raster images. SVG images are basically three different things, they are the shapes that define them, an example would be maybe a circle, a line, a bar, some kind of polygon, we have filters and these add special effects maybe a blur or a drop shadow and then we have gradients. Gradients are another way to paint your SVG object using either your linear or a radio gradient. For this course we will actually won't get into too much details about filters and gradients since SVG objects, which are HTML objects, can also be styled using CSS properties and that's mostly what we'll be doing using D3. Let's take a look at how to actually draw a shape using SVG. First, it's HTML so all we need to do is add an HTML tag for SVG that gives us a place to draw an image and then we give it a tag to draw an image. Here we have a rectangle and it's going to be a width of 50, a height of 200, and it's going to have a style of fill of blue. If we wanted to do this in D3, it's very similar. There's a couple different things here and we'll get into the details of that syntax in a minute, but basically, as you can see, we're appending an SVG objects, we're adding an SVG object, then we're adding a rectangle object and then we're setting some properties of that object. Both of these output the exact same thing, a blue bar. So the very basic way of drawing with SVG is to add HTML tags and D3 is basically a way to do that using JavaScript and of course when we're using JavaScript it is a Runtime language running in your browser that allows us to do much more than just simply add HTML to page. We can make calls to other systems and other APIs, we can also pull in data from external sources, and we can do things in a very dynamic way. Now to understand how SVG actually draws things on a page, we need to understand the coordinate system and think about in school when you have an X and Y in math class and your trying to draw something on that type of a scatter plot. We'll we have that here and this is called the view port or the coordinate system is that instead of 0.0 or the very center of our coordinates being in the bottom left, like we normally would draw in math class, our starts in the top left and this is where we would start our drawing at 0.0. As we move from left to right, we're traveling along the X axes and what we have is basically, an incrementing first coordinate and the second one staying at 0, so if we had a 300x300 SVG window here, we would travel from top left to top right and we would going to from 0.0 to 300.0. If we were then to travel straight down, we web at 300.300, the bottom right corner of the biz and then we can go all the way back to the left and 0.300. So X and Y, just like we normally understand them, however, the origin of where we start, instead of being at the bottom left is in the top left and if we were to draw some shapes here and this is just an approximation, we could have a circle being draw at 20 X 100 Y, and then maybe we'll put another one on there at 40 and 200. This is really important because as we draw things on a page, and whether those are bars, lines, rectangles, or circles, where they get placed is going to be something that is derived based on our data and one of the challenges with that as we'll get into later, when we get into enhancing our biz, is understanding how to take something that is maybe millions or billions in scale and converting that into pixels that can be displayed in a relative fashion. So now, let's get into a demo of actually drawing some shapes. DEMO: Drawing Shapes In this section, we're going to take a look at how to actually draw some shapes. So to do this, what I need to do is I need to have a place to draw those shapes and a way to input HTML and have that be rendered properly. Well, what I'm going to do is use something called jsbin.com and jsbin.com is a way for us to do this without having to use the setup of our machine or anything like that. So if you did want to set it up locally you can on a Mac run something like mamp server or on Windows run something like wamp or anything else that basically renders HTML and JavaScript which should be almost every web server that's available to you, but for this course what we're going to do is use something called jsbin.com to simplify and kind of remove that from the equation and just focus on the code. So I'm going to jump over to jsbin.com right now and we're going to look at how to draw some shapes. Here in jsbin.com, what I have is a basic HTML that it starts with and this is a cool site where you can actually create an account and save them for a future reference etc and in fact in the exercise of this course, you'll actually be able to download all of this code as well. So, we should, at this point, all understand how HTML works so I'm not going to go into too much detail about what we're looking here. I expect that you already understand the basics of an HTML page. So what I'm going to do is start actually drawing something using D3. The first thing to do is to import the D3 library and you can see how I do that here, where I just reference it directly d3js.org and then I give it the library name. I can go download it from that link itself and then Save that if I want to do so, but for this course we're just going to reference the hosted version. So in this first demo, what I want to do are draw a couple bars. So to start what I want to do is give a heading to my first bar, which is just going to be using pure SVG, so I'll just do a h3 tag I'll type this SVG Bar. Now, as shown before, I just want to draw a simple rectangle so I need to start with an SVG tag. Then inside of that SVG tag I want rect, which tells it to draw a rectangle. I'll give it a width, in this case we'll do 50, a height of 200 and we'll give it a style, fill blue. We'll close our SVG tag and we've got one thing drawn. Hooray! We've got something. Alright, and as I mentioned before we can also do this with D3 using JavaScript so let's do that now. I'll give this one another heading, this one will be my D3 bar. Here, instead of just manually drawing it, we're actually going to do a script tag, which is what we need for JavaScript to be rendered, and I'm going to enter d3.select body append, svg append, rct, attr width, 50 attr --- height 200 style fill blue go to ; and then we got to close our script tag and tada, we've got a D3 bar. So, both of these output the exact same thing. In D3 it may look like it's way more complex and you know if this is all I was doing I would agree, it's overkill. As we get deeper into this, what we're going to realize is that since D3 is JavaScript based we'll be able to all kinds of fantastic things using JavaScript to automate and make these much more dynamic. So, I'm going to get into now the principles behind how this works with D3. Let's jump back to our presentation. So what we're doing to start here is we're selecting the body HTML tag. Then, we're going to add an SVG HTML element. Next, we'll go ahead and add a rectangle SVG element, then we'll set the attribute of width, set the attribute of height, and add some style. So that's essentially what's happening with our D3 element there is that it's just generating the exact same code as above. Let's take a look at a couple more examples. In this example, we're going to draw a couple circles and I'm going to open up the JavaScript tab to actually do that. So now, we have on the left our HTML where we could manually input the script tags ourselves, but with jsbin we can actually have a separate console off of that so we can just enter our JavaScript as we go. So first, I'm going to do similarly as I did before, I'm going to draw a circle using SVG. So inside of the body I'll go ahead and add a SVG. This time the SVG I'm actually going to give it a width property, as I mentioned with the view port window before of 50 and a height of 50. Inside of here, I want to draw a circle so the SVG obbj for that is circle and here I actually have to give it some different properties, where is it on the X scale, so that's cx and it would be at 25 whereas it on the Y scale cy = 25, the radius of it, basically how wide or how big a circle is it, just make it 25 and you can see our circle now popped up and we'll give it a style just like we did the other ones, this one will be blue, and I'll go ahead and close my circle, and then close my SVG tag. Okay, so we've got some different properties here and I'll post a couple links so you can actually see all the different options, but as we go, we'll just kind of stick to some basic shapes so we don't get lost in some of the complexities and different attributes that are required to draw the different shapes. Circles are much different than rectangles, it's not just width and height, it's where do you want it to placed within the SVG viewpoint or the coordinate system. As I mentioned in the slide, the system is starting in the top left, not the bottom left, and so where we do this depends greatly on how big our SVG element is. For example, I'm telling it X and Y of 25 and 25 so it's going to put it right in the center. If I change that, and let's say I go ahead and change this to 2, you see that in the output window that the circle moved over to the left. If I were to change that to let's say 0, it'll still stay there. If I were to put it at 50, you'd see it jump to the other side etc, 500, it's completely off the page by now. So that's why I wanted to first introduce the rectangle so you got the hang of building a shape, now we want to take a look at something that actually it matters where you put it. The rectangles are a bit different in that they can adjust to the size of the window, but for the circles, we have to be pretty specific about where they go inside of our viewpoint. Now let's do the same thing with D3. Like before, I'm going to first do a selector, I'll do d3.select. I'm going to select the body, I'm going to use the append operator to add an SVG operator, to add an SVG element. I'm going to set an attribute of my SVG element of width 50 and height of 50. I'm now going to append my circle and give it the attributes of cx50 25, cy, 25, r,25, you can see it pops up now, and we'll style this one a little bit differently, we'll give it a fill of purple and we'll style this one a bit --- differently, we'll give it a fill of purple and good, you can see on the left we have our blue circle that we drew just by manually adding our SVG elements and on the right, we have one that was drawn using D3 and this time we're introducing the concept of this JavaScript window, and this JavaScript basically get auto-ran, you can see there's a checkbox on the right side that'll control that and this is a great way for us to use to debug and for testing all the different things that we want to build. Let's take a look at one more example of building a shape. I'll click Bins, New it will give me a fresh one here, I'll add my JavaScript console and here we're actually going to work with some text. So again, manually we'll do this, we'll add an SVG element width = this time 250, height 250 also, height of 50, and we're going to add some text to the page at x=0 and y=25 and now, when we close that, we'll give it a text here Easy Peasy, close our text tag and close our SVG tag. Okay, similarly how the circles work we have to give our text tag x and y values otherwise it won't know where to put it and the difference here is that we drew something at 250 and 50 and used instead of cx and cy just x and y for our text property. Let's take a look now, how to do this with D3. We start off as always doing select, selecting the body, we're going to append the SVG object, we're going to set some attributes of that width of 250, height of 50. We'll append a text element, we'll set the text value as Easy Peasy, set the attributes here of y, 25 and x at 0, and if we wanted to add a style, could do so, similarly to how we did the other ones. So there you go, we've been able to take SVG elements and draw them by manually entering HTML and we've also been able to draw them using JavaScript and D3 specifically. Now, let's revisit some of the principles and what's actually going on here within D3. D3 Principles So the principle behind D3 start with our selections and this is how with JavaScript we can actually select an element on our HTML page. It uses the same CSS type selectors that you may be familiar with if you're used to doing a lot of web development, so nothing real exciting here, but basically you enter select and the element you want to select. We could do all kinds of different identifiers. We could do things that choose all of an element with a certain type of class or a specific one with ID's etc. So it supports all the standard CSS selectors. Then we have our Append Operators. This is when we append things, when we add new element to the page. We in this case we've selected the body tag and then we're going to add an SVG element and in this case add a rectangle element as well. The last one we've seen so far is the Style Operator, where we append some types of attributes and some types of styling to our objects on the page. With SVG, we're generally setting the width and the height creating that viewport and with the elements themselves they kind of each have their own different properties depending on what they are which are defined as I said in a reference guide that I'll point you to at the end this course. Now the way we've drawn this so far is that a new line for everything that we want to do inside of our D3, but it's JavaScript, so as it goes we can actually get rid of all that white space and chain these together. So this chaining syntax starts our basically with white space looking how we've been doing it so far and this is how a lot of JavaScript is written when it's in development mode, but when you actually want to render it on a page, you want to called minified that and when you minify that you bring all together and you lose all the white space which makes it impossible to read, but it does save a lot of space and makes things load faster. So, that example there could be done exactly without whitespace in the bottom example. For most of this course, we'll be using whitespace, just so it's easier to read and it's easier for us to get our head around and then towards the end of the course we may do some things where we stop doing that just to simplify and condense the amount of code that we have to look at. Now let's take a look at drawing with actual data. DEMO: Drawing with Data Now let's get into a bit deeper of the rabbit hole here with D3 and actually use data to draw something. I'm going to switch back over to jsbin and in a new jsbin here, I'm going to start by introducing the concept of variables. I'm going to start with one called w where we're going to use this to set the width of our SVG object, then I'll have one for height, that is 100. We're going to set the padding at 2, which will give our bars in our bar chart that we're going to building here a little bit of space. Now we're going to create a dataset and in this, this is going to be an array, for now we'll just be pretty arbitrary about it 5, 10, 15, 20, 25. Now instead of just manually or explicitly creating the SVG, we're actually going to assign the new SVG to a variable and then we can actually perform methods or functions using that object later on. So we'll start by saying var svg = and now we would do the same thing we would normally do, d3.select body, append svg and for the attributes of width we're actually going to use w and height we'll use h. So now, we have an SVG. Next, let's add some rectangles to this, svg.selectAll rect data, which is a new operator we haven't see, dataset passing in our data set from above, Enter, and at this point what this means is it's going to take all the rectangles and it's going to create new ones for all the different elements in our dataset. There aren't any on the page, so we use this enter operator to create them if they don't exist. Next, we'll append the rect and we'll make it x of 0 for now, y of 0 for now, width of 20, and height of 100. So we actually have a bar being drawn and with this, what's interesting is it's actually drawing 5 bars, but they're just all directly on top of each other. If I go to Inspect this element, you can see what I have down at the bottom here, I actually have 5 different rectangles all being drawn. So I have an SVG that's been created and then I have my 5 different rectangles. So it used our data and magically it drew 5 rectangles. It drew one for every piece of data in our array. Pretty cool. So how do we actually move the bars based on the data and size them and all that, now is when the rabbit hole gets pretty deep. So the next things we're going to do is instead of the x attribute being 0, meaning it's going to start at the left of the x axes, we're actually going to use the data to position it. So I'm going to get rid of this 0 here and it's JavaScript and the way JavaScript works is that you can introduce functions in the middle of your code, so you don't have declare functions all separately, you can just put them right in the middle living inside of your code, which is how a lot of JavaScript is done and if you're not familiar it can be kind of confusing, so I'll try to take it slow here and explain this as we go. So we're going to do function, we're going to pass in d and i, we'll close that, we'll give it a curly brace, so now, we're actually going to be creating some logic inside of our function. We're going to return i *21 and we'll close our curly brace and there you go. So what we're doing here, we're taking i, which is the index of our dataset and we're returning that *21, so the first one is going to be 1* 21, which gives us the x position, where is it going to start from and then the next one will increment i and it'll give me a 2 and then it should give me a 42. So d in this case is the dataset that's being passed in and in D3 we can reference that because we've defined it up above. Later on in here, we can actually reference this just as d and i being the position of the data element as it appends these rectangles. If I were to go and inspect the element now, you can see I have them starting at 0, 21, 42, etc. So all it did is we dynamically took data, we actually didn't even use the data this time we just used the array and drew a separate rectangle and changed the x coordinates based on the position in the array. So that's step 1, now let's get into how to adjust the width of them. Right now, the width of the bar is static and it's set at 20, but if our viewport was much bigger or we had a ton more elements in our dataset, then they may disappear, they may go off or outside of our window. So instead, what we should do is we should set a dynamic width. So to make the position of these bars dynamic, so that they're evenly spaced throughout our SVG window here, which is 500 pixels wide, what I need to do is instead of just multiplying by 21, I'm actually going to use the data set and evenly divide those out. So I'll take w which is the width of my dataset above and we're going to divide that by dataset.length. Now, if I were to scroll, you can see that 500 is wider than my screen is here, but they're all evenly spaced. I have 5 evenly spaced bars. Now what if we wanted to adjust the width? We'll first what I'll do is I'll adjust the size of our viewports so that's something that we can see the entirety of, and then I want to go down to the width property of our rectangles and use the data again, this time to change the width so it fits the entire size of our SVG. Here I'm going to take w as the width of it, and I'm going to divide that by a dataset.length and I'll subtract the padding, so I still have the padding in there. So now, as you can see, I have 5 evenly spaced bars that take up the entire width of our SVG. If I inspect this element you can see here that it's dynamic, it's 38 and that the position is still being dynamically generated as well. If I were to delete one of these elements, you can see now I had a 4, if I were to add a bunch more, it would be the same thing that I would content to auto-adjust based on the size of the window and the length of the bars. So we're getting there, where things are coming together and being much more dynamic, we're still not actually using the data too much, but we're just using our array, now let's set the height of the bars. So here we have them just auto-defined at 100 and instead we're just going to try to use the dataset itself, so I'm going to change that to function, it's going to take the dataset, I'm going to say return d, close it, indent this a little bit, and there you go. So wait a minute, what happened? Well, as we remember before, 0.0 is in the top left, not the bottom left. So we can't actually just simply return the value and use that to draw the height of the bar. This is somewhat of a dilemma, and honestly, SVG does not care. This is something that we have to deal with and it's been accepted as the way that the coordinate system works, so in order for us to do this, we just have to do a few more things to our calculation here, to get it to basically switch those bars down to where they're in the bottom of the viewport instead of the top left. First is to adjust the y value to start from the bottom. So for the y value, what we want to do is instead of being 0, which is at the top left, we actually want to say the height - datavalue. So let's see what this looks like (typing) and so we take the height of the viewport and we subtract the value from our dataset, d in this case, and that actually converts it to where it starts at the bottom of our viewing window. If we were to adjust the height, we should see this bar change positions on the page as well and if we wanted to adjust these a little bit we could just multiply the value x4, (typing) and there we go. We have a nice little bar chart, we're actually using the data, and in a couple different ways we have the position, we have x, we have to adjust the position for y, so that the bars start at the bottom and we're adjusting the width of them to be dynamically filled based on the size of our SVG, as well as we're adjusting the height. If I were to change this now to 300, you'd see that the bars got a little bigger there and similarly if I change it to 100, they get a little skinnier. All of these things here are the foundational pieces we need to know about in order to use data to draw things our page. Lastly, we could really play with this, we could add some color and we could bring it all different ways just like we did with the styling before, except now we could actually use the data to draw that color, so go ahead and explore, have some fun with this, post your results on the discussion form and I'd love to comment on them and see what you've found using this basic tutorial on just getting started here. Let's take a look and recap what we've learned in this module. Summary In this module, covering the basics of D3, we started out by looking at the origins. Where did D3 come from? What was the motivation behind it? We took a look at SVG objects, what are they, why do we use them and when did they really come into power with HTML5? We took a look at drawing shapes using standard SVG techniques, simply typing HTML to the page and seeing a shape being drawn for us. We also looked at using D3 to draw these same shapes. Then we got into the principles of what's behind D3, the selectors, the operators, and how we can actually use it to generate those SVG images. Lastly, we took a bit of a deeper dive into how to use data to actually draw something with D3. Basic Charting Introduction Hi this is Ben Sullins and welcome to this module Basic Charting using D3js. This is the second module in our course and in this one, we're going to look at basic charting. We're going to start with building a bar chart. Now we built a bar chart in the previous module, so we'll continue from there and we'll revisit that example, so if you didn't catch all that you won't be left out. Then we'll look at coloring the bar chart. How do we actually use data to change the color of the bars? We'll talk about adding labels and how do we do that? Then we'll get building a line chart. Line charts are a bit different than bars in how the shapes are rendered and how we place them on the page and then we'll look at a scatter plot. Scatter plots are a real great way of comparing multiple measures, profit versus sales for example and seeing how things cluster together and where the outliers might be. DEMO: Building a Bar Chart Now let's get started by building a bar chat. First, I'm going to jump over to jsbin.com and we'll get back into our editor where we can start entering our D3 code to build some charts. Here in jsbin.com, I've setup a template that I use that gives me a title and it also includes the D3js script. Let me expand this a little bit so that you can see it and this pulls directly from d3js.org, it pulls in the minified version of the library and this gives us all the functionality we really need to get started right now. So make sure that when you set this up that you include this script in the head of your HTML document and then I expand out this JavaScript pane and I kind of hide the HTML because at this point there really isn't much else we need, we're going to do everything here in JavaScript pane itself. Okay, so let's get started with building a bar chart and we just went through a couple examples of this in the basics so we'll continue on from that and we'll rebuild the chart we just did and then we'll start to enhance that. The first thing to remember is that we should set a window to draw our shapes within. So to do that I'm going to create a variable first one's going to be called w for width and I'll set it to 300. Then I'll set one for h, I'll set this to 100. These are what we'll use later we'll use later in the functions to actually dynamically adjust the size of the bars and the position of them inside of our window. These themselves, could be dynamically pulled in based on the resolution of your screen. This is JavaScript after all, so there's a lot of Runtime type functionality that you can do to dynamically change things and adjust them on the fly. Once we have our height and width set, let's setup some padding. The padding will be used as a way to give the space between the bars. Next, let's do something with the dataset here. I'm going to create an array, which I simply do with a bracket and for right now we'll just punch in some basic numbers just like we did before and then after we have our bars being drawn, we'll adjust the dataset and see how it reacts. Let's create an SVG on the page, so I'm going to say var svg =d3.select body. I'm going to append an SVG (typing) and I'll set the attribute of width to w and the attribute of height to h. So now, we have an SVG, notice I'm doing some chaining, so I'm not having everything be on a new line because we already know what this line is going to give us, so I'll only put those attributes on a separate page because we know that those are actually the ones that we need to work with. Now like before, we're going to go ahead and add some rectangles here, so I'm going to say svg.selectAll rectangles, the data is going to be from our dataset up above. We're going to Enter, which is going to append those if they don't already exist. What it's going to append, it's going to append the rectangle, and we'll set the attributes here. We're going to add that function like we did before, which pulls in d, which is how we reference our dataset inside of D3 once we've added it using the data operator and here we're going to return i * w / dataset.length, that's going to mean that the x position of each bar is going to be dynamically set by which order it falls in inside of our array, which is our dataset. Let's set the y function (typing). This one is just going to use the d itself, it doesn't need the actual index of where the array element is and if you recall, in the top left, is where we start in terms of our coordinates, so we actually need to take the height of our SVG, which is represented by h and subtract the actual data element itself, the size of the bar, in order to properly place it at the bottom of where the SVG starts instead of at the top. Next, we'll add the width, and this one is w so the width of the entire SVG divided by the length of our dataset meaning how many elements are in it minus the padding. Next, we'll set the height of the bar, which the height of the bar is going to be dynamic based on our dataset (typing). This one is simply using what was provided to us in the dataset itself and once we enter that there you we can see have our bars. Now we didn't size them appropriately because the height of the bar may not actually fit properly within height or our view window. We could take these and just multiply the y and then the height *4 and that would actually fill up the whole window there. Now we know that because we've manually of explicitly set our data values here. In the next module, what we'll actually get into is how to dynamically set that because for now if I were to change, so let's say this one to 50, what would happen is that the bar is actually taller than our window. So if I wanted to adjust that I would have to actually go down, make the height of it much bigger so you could see the total size of the bar. So be careful about that, just doing *4 right now is fine because it just fills up the page and we know that the domain is from 5 to 25 meaning that *4 is going to give us exactly 100 pixels, which happens to be the height of our SVG window. Now let's take a look at enhancing this a little bit with some color. DEMO: Coloring Your Bar Chart Now like before when we were styling our shapes, when we were just building them manually, we have an attribute called fill that we can use for our bars. So it's pretty simple, I'll just go ahead and add another attr here, I'll call it fill and let's make these blue. We need to get rid of our old ; there and give it here, expand this out so I don't need to worry about seeing the HTML. In fact, we'll just go ahead and hide the HTML pane, so we're entirely dealing with the JavaScript and the output here. So that's cool, so now we know how to add some color to them and that's one way to do it. Now we could also use all the web friendly colors, we could use the RGB codes, or the hexadecimal codes, either way, but what if we wanted to actually color these bars by something? Well, just like before we could use data to do that. So let's take a look at how we might do that and so to set this dynamically, we'll basically create a function, just like we did before, that references our dataset and in here it's going to return an RGB code. We can play around with this and maybe the first one what we'll have is d * 10 + and then we get the rest of our RGB code (typing) and now we have red bars. So the RGB codes, you can look up, there's a really great website colorpicker that has a bunch of these and you find the right ones, but this is way now to take our data and use it to actually color or stylize our bars. Similarly, if you want to change this to a different RGB mode, we could Cut this guy out, it'll be black of a second, now I'm green, and I think the last one, if we were to add that there, it'd be blue. So a very basic way of doing it. Again, we may later one look at the advanced ways of identifying a color pallet and actually choosing the color very purposely instead of just having a gradient type scale. Now as we go along here, you may think that, man this code is getting really messy, I have all these calls to this attr function and I keep having to repeat all this stuff. Well, D3 does offer some simplicity or ways of looking at this more cleanly and that's called a multi-value map and it's almost like passing in JSON to our functions, so that it can parse the entire thing without having to repeat each function call, and JSON if you're not familiar with its JavaScript Object Notation, it's a way of handling data on the web that is commonly accepted as kind of a standard for a lot of the new web apps that you might be using. So instead of making this attribute call 10 time or however many time, we can actually do it just one and pass in sort of an array of all attributes that we want to set and to do that, let's go up to the top one here where x is and in here I'm going to start and I'm going to put in a curly brace there, which is going to tell it that I want to now use a multi-value map and here I can just say x: give it that same function, I'm going to put all this on the same line so you can see it, so it simplifies it yet again. Now for the next one, I just bring it over y:, my function and I'll just clean up the rest of these here. We're now able to pass in multiple attributes into a single call and this is the method that we'll used going forward so it's just a lot cleaner, but when your first learning, it makes sense to explicitly call them all out, that way, it's very specific about how to do things, and the last thing I'll how you here on coloring the bars is how to add a function to use to add sort of a KPI indicator or a very specific choice of color. So to start, up here were my variables, let's add a function called colorPicker and this will accept v as a value, put my curly braces in there, and I'll just have a simple statement here. If v <= 20 (typing), return hexadecimal code, which is a dark gray else if v > 20 return ff0033, which is this kind of bright red almost pink, something to draw our attention and now instead of where down here I have my function to get this RGB code, I'm actually going to call a function and we'll say return colorPicker, which is the name of our function, and we're passing in d, which is our data from our dataset that we're iterating through, and as you can see there, the output instantly updated, and it has this dark gray with this bright pink indicator that's telling us, look at me, look at me. If you're wondering about the use of color in charting, I'll skip over most of it here and I will get into the philosophical pieces of it, but if you're interested in that, I do have two other courses out there on Data Analysis Fundamentals, as well as Business Dashboards, which really give the details of the visual presentation of things. The more conceptual examples. Here we'll just assume more the functional pieces of how to actually make color appear or how to use color. Similarly, in our colorpicker here, we could have any key performance indicators and this number could be coming in as something like sales or profit etc. Okay, so the visual aspect is one thing, but we don't actually know what the numbers are yet, so let's see what we can do to add some labels to this. DEMO: Adding Labels In this section, we're going to look at adding labels to our bar chart. I'm going to jump back to what we've been working on in our jsbin and in here we'll make a it a little bit more interesting, let's just add some more data. You can see that our data again is doing this kind of scaling to whereas we add data elements it automatically adjusts to the size because of our functions down below and you could see the coloring is still working using our colorPicker function. Now if we wanted to do something here where we added the labels, all we really need to do is add another SVG element of text. We're already adding the rectangles, now it's time to add the text, which will act as our labels. Here again, I'm going to svg.selectAll this time we're talking about text, we want to pull in the dataset again. We use Enter to make sure that it'll add the elements if they don't exist. We're going to append text, the text of it is actually going to be the data itself. Now we'll pass in our attributes here, just like we did before we're going to use a more complex way of doing this that actually simplifies the display which is nice and I always like to get all my open and closed parentheses first that way I don't get a ton of errors and stuff as I'm going and I forget one somewhere along the way. So here, we're going to say, text-anchor middle and the x and y functions are going to be pretty similar to what we had before because we basically need to put them in the middle of the bars, so the x is going to be a function, which is going to be d and i just like the other one. This time though, we're going to do return i set the position, w / dataset.length just like the other one, we'll indent this a little bit, give these a y position, which matches what we have already in the bars. The text-anchor 1 because it has a dash in it needs to actually have quotes around it so you can see what it did there and the Excel value is kind of funny, but we have to do a little bit more because we have to adjust where it lives within the bar essentially. So when I fix my quotes there around text-anchor telling me where to go and I fix my formula to match what I have up above, you can see the numbers appear. One of the issues might be that their falling outside of this, so let's go ahead and increase the height just a little bit and you can see now my numbers pop-up for those top bars. Remember, we're doing *4 so that means at 25, it would take up the entire height of the SVG window. So let's add a little bit on there so it will actually be able to see those numbers. So it's still not centered properly, so instead of just doing exactly where the x is, I have to add a little bit more of a calculation here to divide where it is and what this is, is to figure out to basically account for the padding and add that to it, and then find the middle. So to my function down below I'm going to hit x I'm going to do +, the width / by the length of my dataset - padding / 2. There you go, so I have now the labels basically sitting right on top of my bars and if I wanted to I could also play with where these start from, notice the y is sampling using the same function that we used to draw the baseline for the bars. I could add something here based on maybe the font size to where they fit inside of the bars like that. Now if I were using color wisely, I may have to have another function that basically says if it's inside of the bar, also figure out what color the bar is and then that may adjust the font color. So for now, we'll just leave it like so and if I wanted to get more stylized on it, I could add a font-family. Remember when you have a - in it, you have to put the actual quotes around it and we need a comma between each one, so I have a sans-serif font, which is a little bit cleaner. If I wanted to adjust the size, this is basically just adding CSS properties now. Something like that, and again, we could change the fill of this object if I wanted to make them white like that. Okay, so, we've look at how to add labels, it's essentially adding another text SVG that follows basically the exact same x and y as our bars, the difference here, we're going to manually adjust them and stylize it little bit differently. I could add more of these and place them below, if you wanted multiple labels like percent + or -, but this should illustrate at least now how to draw a bar, how to color the bar, and how to label the bar. Labeling essentially will be the same for almost all the other objects. You follow a similar pattern as the shape you're drawing and you adjust it based on where you want it to appear on the top, on the bottom etc. DEMO: Building a Line Chart Now we're going to take a look at building a line chart. Now a line chart is a bit of a different thing because it's not just a shape it's a mark that has a path along our page. So, let's jump back over to jsbin and here in jsbin I have my standard template which has D3 already embedded and I have a title and all that, so what I'm going to do here is take the same data we have been working with and now this time instead of creating an object I'm going to create a function that's going to draw a line for us. I'm going to call this lineFun and here I'm going to say d3.svg.line. Now line is an operator that generates some really interesting SVG HTML for us to draw a line. So what I'll do is I'll run through this function and then we'll take a look at the output and see what it generated and then I'll explain a little bit about what that is and why you'd never actually want to try writing this by hand and you'd rather use these functions to save yourself a lot of typing. For our line, what we want to do, is we want to add an x the function, we're going to use the data that's being passed in. For y, we'll use sales and now I'm going to say interpolate and I'm going to tell it what type of way to do that and that's linear. There's a lot of different ways and they will give use different types of line charts and I'll show you what those look like after we get a basic line built. Okay, so this is basically a function that we're going to use and we're going to pass in some data and it's going to draw a line for us. So let's get back to our normal mode of operating here and I'll create my SVG. Okay, we've got our SVG and now let's make a viz. We're going to append a path this time and this is the special type of SVG object, which allows us to draw things using data. In fact, it could actually draw any number of things including rectangles, and squares, and circles, and all the other shapes that we have already seen, however, it's really easiest to use this one when we want to draw something that doesn't have a predefined shape that's known to use through any of the other commands. A good example would be if we wanted to draw a map, the borders of a city of state or country aren't squares and circles, but they're polygons of some sort that have a lot of different angles and that's where the path essentially takes the data we give it and draws that shape for us, which is how it saves us all that time. Just like the other ones, I'm going to add some attributes here. Now I'm going to call my function and I'm going to give it a monthlySales. I'll give it a stroke which is just a way of coloring the line, give it a stroke-width which is essentially how wide you want the line to be and fill we'll say none and as you notice, we now have a line. So what's happening here. Let's take a look at the actual HTML that it just generated and to do that I'm going to right click and choose Inspect Element. Now I'm in Firefox, if I were in Chrome or Internet Explorer, you should have similar options here to get into the details of what's actually being drawn and the inspector here is different because JavaScript updates things in Runtime, which means that the original HTML, as you get when you do a view source on an HTML page, has been modified now and the new version is the one that we need that actually will show us what has been generated or the current state of the HTML document and as you see that down here, I have an SVG, and when I get in there I have something called path. Now remember before we were drawing rectangles and circles and all that, and what it's actually doing is it has this d operator and this d operator has all these different coordinates and what those coordinates are those are the points in the line and it's telling it exactly how to draw from one point to another so it actually creates that shape. Now like I said, this could be used to draw something like a square or a circle, or a map, or the shape of a state let's say, for this case, we're just going to use it draw a line and the thing that you need to remember is that we create a function to draw the line that uses the data we're sending in, we do the standard thing where we create an SVG where we will draw our line and then we have a function here that draws our viz, our visualization, and instead of appending a rectangle or a circle, it appends a path and it calls that function we created using that data that has been loaded. So this is the basics of how to draw a line. Now let's do another line and this time we're going to add some labels and make things a bit easier to see. To do this I'm going to go back to the data here, I'll hide my Inspector, and I'm going to update this what you'll find in the line-2.js file in the exercises of this, and it's essentially the same data, just a little bit different values here to make the line a little bit bigger for us to see. So I'm going to Copy and Paste that in now and with that there you can see the line looks a little bit funky, it changed its shape quite a bit, as the values did, I also adjusted the size of our SVG, so that we can see things a bit more clear, and so the first thing to do is to get this to align correctly just like we did the bars. I need to take my y value and subtract it from the height and when I do so, it actually correctly draws the line instead of drawing it upside down like we saw before and to make this fill up our space a little bit better, we'll *3, and now let's go down and add some labels. So I'm going to create some space down at the bottom here and similarly I'm going to give it a name, assign it to a variable called labels. This one's going to take the SVG, it's going to selectAll text. If you remember from the previous module, where we did this to draw text on the page it's essentially what we're doing here. For the data, we're going to take the monthlySales. We use the Enter operator, then we're going to append text. So at that point what it's going to do is it's going to look for all the text objects in our SVG and it's going to look at the data and monthly sales and then it's going to assign the data to those text elements, but if they're not there it's going to append them. Now, for the text itself, what's being displayed, we're going to use a function and we're going to return sales. Now let's set some attributes for this. We need to tell D3 where to put our labels, so we'll have x and it'll take a function and it'll return the same that thing we're doing up here as to draw our line, this way it'll show up in the exact same place. So I'm just going to Copy that, Paste it here. For the y value, let me hide the HTML so you can see this more full screen. For the y value it'll be the same thing as we're doing up above when we draw our line and you can see it start to build out here. Now I'm going to adjust exactly where these appear on the y axis because I want them to be a little bit off set from where the other ones are. So I'll just - 25 from the x axis there and it'll kind of make them appear on the left side, so it's giving us a little bit more margin. As we get into the more complex examples, we'll have dynamic ways of doing this to where they always appear correctly, but for now, we're just kind of getting the feel for this, so we'll just hard code a lot of this stuff. Now we can put in some styling, font-size 12px, font-family sans-serif, things are looking a little bit cleaner, fill I like this darkish gray. Now, text-anchor is going to tell it where to appear, so we'll put text-anchor at the start, so you can see it actually looks at where the x and y are and in that space, does it align with the left, the middle, or the right and that would be the start, middle, or end. Let's set the line height here or the line size. Things are looking a little bit better. Now we're going to use our line-end so we've done a lot so far. We've seen how to add labels here and we can adjust that and we can play with the styling, and what if I just wanted to highlight the different lines at the endpoints? Well we can do that using the index. So let's change the font weight, which is how you set Bold or narrow for example and we're going to use a function here, which takes our data and also the index, the position, and for this, I'm going to have an if statement. If i = 0 or i=length of my monthlySales array -1 because it will essentially find how many elements are in the array and then the counting is of a different index so, one of them starts at 0 the other one starts at 1 so I have to -1 so they're on the same scale, and basically, at this point what it's going do is going to say, if basically it's the first data element or the last data element, now let's do something, and I'm going to return bold if it is, else turn normal, and there you have it. So as you can see 100, the first data point is bolded, and 130, the final data point is also bolded. So this explains a lot about how to draw lines. We take a dataset, we create a function here that will draw our path for us and then it's very similar to how we do all the other ones. We have an SVG and we append the object that we want, path, and then we use our function to interpolate this across our dataset. You use this line function which generates all the points necessary in a path to make it actually appear as a line on the page. As I mentioned, we can play with the different types of lines. So right now we're using a linear one which from a visualization standpoint, I typically recommend most business settings, you don't want to make things skewed or decorative, so I would just keep this as your standard, but there are many others, and so one of the ones that I can do is called bspline and it uses this function called basis and now we could actually go in here and adjust these a little bit or see exactly where things want to appear and we could mess with that, so again, a lot of options here and a lot of ways to do this. Here are just a few and as we get into the more complex examples, we'll start doing more things and generating this more dynamically. Now let's take a look at another chart type called a scatter plot. DEMO: Building a Scatter Plot In this section, we're going to look at how to build a scatter plot. Now first, I'll just talk about what a scatter plot is. It's a good way to put dots on a page to see where things cluster or to identify outliers. Typically, we're looking at two measures, let's say sales and profit and how they relate to each other and maybe where are the outliers or where are the clusters of sales and profit combinations. In this example we're just going to continue using the dataset we've built because we're already familiar with it and we're going to put dots on a page, which are similar to what we see in line, except it's just not connected. So imagine the first time you drew a line chart you essentially put the dots first and then drew the line. We're essentially now just going to do that except we're just drawing the dots, we're not connecting them to build the line. Let's jump back over to jsbin and I'll start off where we left with our line chart and so for here what I'm going to do is I'm just going to wipe out everything that creates the chart and just leave our data. So we're essentially working with the same data that we were before and to create it, it's very simple. First, we create our SVG, just like we've been doing, d3.select body, append an SVG with the attributes, width of w, height of h, that gives us an SVG. Now we essentially just need to add dots or add those circles where the points would be. So I'm going to call this dots =svg.selectAll circle is the shape we're looking for, the data is going to be monthlySales, we'll hit Enter, then we'll append circle. We'll give it some attributes, cx instead of x. We'll use our data by calling a function and we'll return d.month*3, cy also a function this one's going to be return the height of our SVG -sales. The radius of our circles we'll just put it at 5 for now, and the fill of that dark gray that we like. Let's close out our attribute and viola, we now have the basic scatter plot. If I right click on this and take a look, Inspect Element, I'll drill down into my SVG, you can see what's doing here. So it's basically creating a circle for every data point in our dataset and it's going to use the attributes I give it. Now let's do something a little interesting and let's set up a key performance indicator that we'll use to color the different dots. Now key program indicator could be anything, but in the context of a business this is something that you would use to identify qualitative attributes of your data, not just quantitative like the numerical values or where the fit in a line, but the qualitative attributes. Are they good or are they bad? How do we define that? Usually, it's defined by the business in some context, but we can actually code that into our visualizations to make thing a little bit easier for users to understand. So I'm going to create a function here, I'm going to call it salesKPI. It's going to take our data, and it's pretty hardcoded at this point. If the data is >250 return, I have a nice color here of 33CC66 else if d<250 return the gray that we love so much. Okay, so we have a function now that will actually color code our data points or our circles and we need to change the fill down below from the being a static color to calling our function. So what I'll do is I'll create a function call, and I'll return salesKPI and I'll pass in the sales value and there you have it. So now, those points that are above or equal to 250 in sales are this bright green and the other ones are this dark gray. So when you look at this, what we're doing is we're coding into our data, a way for users to easily consume the data, we're highlighting the data points that are following some logic, in this case it's rather arbitrary, but you similarly could, and as we will later on in the course, add in more complex logic derived from other datasets and other rules. So this is just one way we can make our visualizations more meaningful to our users by using color , as well as the position on the page. Now what I'd like to do is add the quantitative attributes here as well by showing our hiding labels based on this KPI and for this we're going to create some more functions here. So we're create a function called showMinMax, it'll take our dataset, the column in our array, the value, and that type of label we want to show, so it's a little bit complex and there are many ways to do this. This is just one fun example that I like use when first explaining how to make things a bit more interactive and as we get into this later, we'll actually show how to give the user controls for this. So here, I'm going to have a couple variables the first one is going to be max and I'm going to say d3.max, which is a function that we can use, we're going to take our dataset and then we're going to have function, which is going to return whatever column is being passed in and we've learned that we can reference things with a dot as in d.sales, we can also reference them this way and this way if there were a space or any other kind of funny character in there, which is not recommended, but if there were, our functions would still process it correctly. Okay, so we have our max, which will take our dataset and find the max of the array essentially. Now we'll do a min, now we'll have a simple if function, if the type = minmax && val = max or val = min, then return val else if type = all return val. Add our final curly brace and we now have a function. Okay, so what this is going to do, we're going to call this when we want to add some labels. It's going to find the min and the max and then if the type equals the min and the max and the values are those min and max values, it'll return them, otherwise if the type is all it'll them otherwise it won't return anything. So we have two options here, we can say show all labels or show just the min and the max. Now let's go add our labels here, so I'm just going to drop this down at the bottom like we have with all the other functions, svg.selectAll text the data is going to be monthlySales, use our Enter operator, and then append text. For the text itself we'll actually call our function, so first we do the regular function call we use. Now we say showMinMax, give it our dataset, monthlySales, sales is the column we want to look at d.sales is the value and for now let's do min max. So that's our text function. Now let's do some attributes be we need to tell it where to put them and what they should look like, x function it's going to be very similar to what we did before, return d.month, we're going to multiple this *3 and then if you recall we had an offset off set -25, we can play with that later and see how it all lines up. For the y, we'll do very similar function d return this time the height-d.sales and you can see our labels have now appeared. Let's style this a little bit more, font size 12 px, font-family sans-serif, fill let's use that dark gray and we'll set the text-anchor to start, and there you go. So we now have a dynamic function which will allow us to manipulate what type of labels we'll see. We're playing with colors here and we're adjusting those to show our users the qualitative side of things and we're using our labels as a way to show the quantitative side of things. So maybe give it a little bit more space and what if we did all instead of min max? Now they all showed up. So imagine we could build a control, and we will later to do things like this, that would allow users to adjust this here and as you see it updates in real-time. So our data visualization can be highly interactive and that's really the nature of why we're doing this. If all we were doing is generating a single image, that's much simpler and takes a lot less coding to be honest, but what we're trying to do is make things interactive and fun in ways that people can touch and feel the data, so that they can really get connected to it and try to answer questions and learn something hopefully. Now let's recap what we've done in this module. Summary We started off by looking at how to build a bar chart and then we looked at coloring the bar chart and adding some labels. These are the common paths and methods that we use to manipulate a lot of our visualizations. So starting out with the bar chart, made a lot of sense because it's a very basic shape that we can all conceptualize. Then, we took a look at a little bit of a complex thing with a line chart and in the line chart it got interesting because we learned something new, something called path, which allowed us to draw polygons or in this case a simple line, but trust me that'll come in handy later when we try to draw things like maps that are more interesting, and lastly, we looked at building a scatter plot and some of the different ways we can use functions to dynamically show and hide labels, as well as adjust things by color. Working with Data Part 1 - External Data Sources Introduction Hi, this is Ben Sullins and welcome to this module on working with your data, this is a two part module and the first part we're going to look at external data sources. Checking in on the course, we're right about midway where we talk about working with actual data getting into the very practical uses of D3. To start we're going to look at external data sources, just the overview of it and what the different types are. Then we'll look at CSV data actually doing demos of working with this, and then we'll move onto JSON data. CSV and JSON are probably the two most common formats you're going to come across when you deal with data in JavaScript and so I wanted to cover those in depth and then in the second part of this module we're going to go into calling APIs, Application Programming Interface to get real live data from the web. Let's get started. External Data Sources Let's talk about external data sources. Thus far, all the data we've used is data we've hardcoded into our scripts that is we've manually typed it in and used that for all of our visualizations. We'll most likely in the real world what you're going to get is data that comes to you from a different place. It could be from a web resource like an API, or it could be a file you download, or it could be something that maybe a data engineer hands off and lands on your web server for you to use. In any case, real data most likely comes from somewhere that wasn't hand typed by you and when we look at these we have two different types, one is files based, these are files that would land on your web server and you'd reference them locally as you would an image file and we have all different formats here. We have CSV which is Comma Separated Values, TSV which is Tab Separated Values, JSON which is a JavaScript object notation probably the most common one on the web right now. XML is another popular one to be used on the web. We have basic text file that could be delimited or fixed with, and then we have HTML files, sometimes these contain data themselves and we can pull that data into D3 and you do things with it including building visualizations. Then on the right side we have our more dynamic data sources, GET and POST are types of requests that we can issue on the web, this is how most communication happens between different websites and even internally to GET information or POST transactions etc, and then we have AJAX, which is asynchronous JavaScript in XML, something you'll never need to remember, but something that you'll hear a lot of. Another term for this is xhr and xhr is a reference to a type of XML object that we create, which allows us to basically do these transactions between server side, something living in a database let's say or some application logic, and the frontend where people are actually looking at stuff on the web. AJAX is the method that allows us to actually make those transactions seamless for our users. So if you go to a website and you log in and things just magically change, the page doesn't refresh and take you somewhere else, that's most likely an AJAX call and we can do that here and use that method to get data and make things highly interactive for our users. Now let's take a look at working with CSV data. DEMO: Working with CSV Data Working with CSV data is an interesting thing that we need to accomplish in order to then move onto the more complex data types. So we're going to start with a simple example here and work more into a complex one. Now up to this point, we've been using jsbin.com to build all of our visualizations, as a real easy way and universal thing that we can all dive right into and not have to worry about the setup of our web server or any of that, well now that we're talking about working with external data, we actually need to have a server or something that we can reference for local access to a file. So for that instead of using jsbin.com I'm going to use something else and this is called Cloud9IDE. Now Cloud9IDE integrates with GitHub and a lot of other interfaces where you can actually store your code, you can code together and it's a really interesting way to create these environments without having to worry about too much of the setup to really just dive right in and start coding. So for a lot of the course going to forward, we're actually going to be using Cloud9 to do our development in. So in order use this, the first thing you need to do is go create an account. I go to c9.io, I click Sign me up! We'll use the Free package. I give it my Username Email, click Create free account and I'm done. I've already done that, so I'm just going to go ahead and login and now that I've logged in on the Homepage here it says Welcome Back. I'm going to click Go to my Dashboard. On the left, you'll see all the different projects I have and what you want to do is create a new workspace. I'm going to call this d3js-data-vis-fundamentals. You can call yours whatever you like and we're just going to make it an HTML5. We're going to say hosted, click create, and it'll start processing here. Once this is done, we'll go ahead and open that, I'll click start editing and here I am with my very own workspace where I can upload files, I can create new files, and I can preview them all in our window here. I'll go ahead and close that and so to start what I'd like to do is take some data from the demos.zip file, associated with this module, and I'm going to upload that here and then we're going to do some of the basic things that we've done before like doing bar chart and a line chart, but we're going to do it now using data coming out of an external file a CSV file. So here I am and I've opened up the demos.zip that came with this module and I'm going to go ahead and take a look at my monthlySales.csv. If I preview it down here, you can see it's pretty basic, I have month and sales and then I have a month which is the numeric representation of it, and then I have the sales which are the numbers for that month. So what I want to do is upload this to Cloud9. Let me switch back now and I'm going to right click on my folder and say Upload to this folder. I'm going to go find my file and drag it in here. Okay, now I can even double click on it and I can see the data right here and I can edit that in line. For now, I'm going to close my terminal window here and now I have data up there. Now let's take a look at see what we can do with it and what I'll do now is upload these other resource here by clicking on my Folder, right clicking and choosing Upload, going and finding my D3 JavaScript library and my HTML template and I'll just drag these guys on where it says drag and drop. Once they're uploaded we'll just open our HTML template by double clicking on it. and I already have a little baked in hello world example here, so in order to see if this works I'll hit preview and tada, there it goes. So what we have here is just the basics of getting started using Cloud9IDE and this is what we're going to use for this module, so I'm going to start building these things out and because it's in this environment, I'm able to access these resources locally, which is good because as you'll learn later on when we try to access remote resources there are some challenges that we face. So let's get started here working with our CSV data. I'm going to close this guy. I'm going to Copy this by saying Duplicate and now I'll rename it and I'm going to call this working-with-csv-1. So this is the first example of working with our CSV. I'm going to shrink up my preview window a little bit here, so we can see our full page there and the first thing I want to do is just access the CSV and write it to the console to make sure that the data's being loaded into D3. So in order to do that, what I'm going to do is just like we did before, I'm going to set some variables up, I'm going to give my height of 100, width of 400 this time and I'm going to create a variable to load our dataset in and I'm just creating it here because this is where we'll have a global variable for our dataset. I'm going to get rid of my Hello World example here and the first thing I want to do is I want to say d3.csv and this is a method that will pull in the CSV data and parse it out into a dictionary or an object for us and this is going to be a similar one d3.csv.json.txt.xhr, all these different options to pull in data and there's a reference for this online that I'll post at the end of this module, so you can actually go if you're digging into this and you get lost about which one it is, you can look up and see what the right method would neighbor to call. So for here, I'm just going to give it the name of the file I want, monthlySales.csv and it has a callback so we're going to do a function and there's two things that are returned, one is if there's an error, the second if there's some data. So at the end of this make sure I put my ; in the right place, and here we're just going to do a basic check. So it's going to say if error console.log error, now else, console.log data and then I'll set my ds = to data. So I'm now assigning ds as my global variable, this method that's returning the data. Okay, so now I'm just going to hit preview, but first I want to hit Inspect Element on my Hello World example, which will show me my inspector and if you're in Chrome or Firefox or Internet Explorer, you should have all similar options there and what I'm going to do now is just Save this first off, I'm hitting command C on Windows you'll probably hit Ctrl+C, I'm going to hit Preview and I get something output here, I get Object, Object, Object, Object etc and what that is, is that's essentially an object for every row in my dataset. So that worked, so we have data. We're golden. Okay, now, we need to do something with the data. So the things that's interesting about this is that when we call this method, this is going to run asynchronously with any other functions or any other statements we put in here meaning it's not going to wait. These things are going to run sequentially in order, so in order to do that this properly, usually what you'll do is down in the bottom here you'll type in your code to actually draw your visualization. So if we use the line that we were building before, I can just type that in right here, if you remember it was called lineFun and we're giving it that line function. We're going to say x is a function and it returns d.month-20130001 and we're going to do /3.25 just to adjust the size for this demo here. For y, we're also going to do a function (typing) and this time we're going to return, the height -d.sales. We're going to say interpolate linear and that's our function to draw our line, now we're actually going to build the SVG so var svg = d3.select body.append svg, the attributes are width w, height h, so that gives us an SVG now and we'll make our line chart here my creating a new one called vis and we're going to say svg.append path.attributes, but this time we're going to be d: lineFun and we're going to pass in ds, stroke we'll make it purple, stoke-width we'll make this 2, and fill none for now. Okay, so we essentially put all the code that we normally would have just right up in the script, we put that inside of this call to the CSV and like I mentioned that's because this call to the CSV is going to execute regardless of any of the other functions or anything we put in here, so essentially it's going to execute the call and continue on regardless of whether or not we have something after that that needs that data. So in order for it run in the order that we expect, we actually have to build it into and put that inside of the call. So now let's hit Save and Preview this guy, and tada we have a line and so the really cool thing is that we've just been able to access some external data and pull that in and draw a line using that data all within JavaScript, all within D3. If I go right click on this and choose Inspect Element, you guessed it, it's just like before where we had all those different points using that path object inside of our SVG window. You can see it has a highlight here how it does that. I'm going to close the Inspector Window. As I go through here, we're going to use this and we're going to draw more complex things, so I'm actually going to take this whole piece of code and put that up above into its own function called buildline (typing) and just Paste this in and now inside of here, I'm actually just going to call that function buildline. This way if I had other charts I wanted to build or if I wanted to loop through our data and build multiple lines etc, I can just call that one function and I don't have to rewrite the code over and over or deal with the messiness that will ensue as this thing starts to get more complex. Just to test if this works, I'll hit save, I'll go ahead and hit Preview, and it still works, I still get my line. I'm going to go ahead and Save this now, as working-with-csv2 and in this one we're just going to do as I mention there we're going to get a little bit more complex and this time I want to see some totals. So I'm go ahead and scroll up to the top , let me close my preview window for now so I can see this full screen, and if I hover over this on the left side, you can see I can actually collapse these things to make it a little bit easier to read and this time I'm going to build a function called showTotals, and here we want to use a text object and we actually want to put it into a table. So it'll be pretty straightforward. I'm going to have an object called T it's going to be d3.select body append table. Now I need to get the total that I want to add and there are probably a dozen different ways to do this, this is one that makes a lot of sense to me, so I'm actually just going to follow this where I use a for loop to actually iterate through and just create the total for me. If you're familiar with a for loop, basically, what I'm saying here is I'm creating this variable i, setting it = to 0, and then I'm going to iterate as long as i is > the length of the dataset which is the array that I'll get back from my CSV call, I'm going to increment i + 1. So I'm starting out with i is 0, I'm going to go until i reaches the length or just before the length of the dataset and each time I run through this loop, I'm going to iterate it on +1 to i. Make sure I got my curly braces setup and simply I'm going to say salesTotal += ds i sales and *1 just to convert it to a number, it's kind of a shortcut there. Now we need to add the total and for this we're going to create a new called tr or as a table row and we're going to say t.selectAll tr data, in this case it's just 1, it's just arbitrary giving it a variable to draw a row. I'm going to say Enter append tr, append traditional, which is a cell in the table, and the text of which is going to be salesTotal + salesTotal, we're almost there. Just like the dataset we'll go ahead and make this a global variable and let's set it to = 0.0, gives us decimal feel, so when it adds it up it has the right data type, and lastly, we'll come down here and say showTotals, so now we build a line and then we show the totals, let's save this guy and do a preview and there you go. If I right click on this and go to the Inspector, you'll see what I have is I have my SVG, which is our line and then I have a table with a single row and a single cell of that salesTotal. So the idea here isn't that I show you every little example and how to do every little tiny thing, but how to get your brain thinking of making these things more interactive and more dynamic. You saw in the previous examples how we could add labels dynamically and color things dynamically and here we're doing some very similar basic type things, but this time we're actually using an external data source this CSV. Let's do one last example here and I'll show you how to actually make the data that's being shown a little bit more detailed. So I'm going to go ahead and Save this as working-with-csv-3 and here I basically want to do the same thing except now I'm going to show the average along with the total. So just after I get my total, let's do a new one where we actual calculate the average. I'm going to call it salesavg = salesTotal / ds.lemgth, so I'm taking the total / by the length, so the number of rows in my dataset. I'll go back up to the top and let's add this as a global variable as well (typing) and while we're here let's add another one to contain these metrics and this one is an array, so I'm just going to give it the two boxes there. Alright, so now when I go back down, what I need to do is actually add both of these totals to an array and then use that array in my text table. So pretty basic stuff, add metrics to array, here I'm going to say metrics.push, which is a function to add an element to the array, salesTotal + salesTotal then we'll do metrics.push sales avg and sales avg and I'm going to do a function here which was toFixed which truncates it to two decimal points otherwise it could be a 16 character, 16 digit long number and then down in here, where my data was just 1, this arbitrary thing before, now I'm going to pass in my array which has my metrics, and for the text what I'm going to do is call a function, just like I would if I were doing this regularly in D3. Here I'm going to return d because this array only has a single, it's not a multiple varied array. Okay, and let me run this real quick and after I Save and hit Preview, you can see now I have both of them there. If I were to Inspect this you'd essentially see the same thing just now it has two rows, two tds and the data's being displayed, so I'm using an array here to push out a series of summary statistics along with my data, along with my chart. So we're getting into working with data now, let's take a look at JSON a different method of storing data and one that we're probably going to come into contact with quite often when we're building things on the web. DEMO: Working with JSON Data So in this section we're going to take a look JSON data. JSON stands for JavaScript Object Notation it's a very common language that is being used on the web to translate data around and one of the key reasons why is because it's very easy to read, it's very human readable and human friendly. So let's take a look at what some JSON data looks like and then I'll show you how to work with it in D3. Here I am back in Cloud9 and what I'm going to do is upload my JSON file that's in the demos of this module and we'll walk through exactly what it looks like, and here I have three different JSON files in the demos.zip. I'm going to upload all of them right now because we're going to use them all in our examples here and now that we have those files uploaded, let's just crack open this monthlySales.json and take a look at what we've got here. So if recall the CSV was the typical format that we're used to in working with data in our desktop tools like Excel, where it's just rows and columns essentially. If we open up that same file you can see the data looks very familiar to you, but if we look at the JSON it looks a little bit different certainly more line here, clearly more text so it probably compresses less, but the way this work is it's very readable and this is the most simple version of it. As we get into the deeper side here, we'll actually see how things become more complex. For example if I were to see sales by category, I essentially see how now I can have a one-to-many relationship with the data in my file or I have let's say header information you could call this and then you have details information in a separate array or a nested array within my data. If I had multiple categories, it might look something like this where I several layers contents, category, and then I have my montlySales and then I have my next category. So JSON's very readable, it's very easy just to scan with your eye and see what's going on. If you'd like to take a look at some other data examples out there, there's a lot of them data.gov for example has a great repository of data and on there you can search for JSON data and things can get really complex, but for right now, just think of it as another way of storing data that is very readable and very easy, and when we actually translate this into D3, like most programming languages, we have a real easy way to convert this into an array and then work with it just like an array. So I'll go ahead and close these files here and leave this one open, this is monthlySales.json, this is the one we want to work with. So I'm actually going to go back to my-working-with-csv1 and I'm going to open this guy and save this as working with JSON1. I'm doing this because it's quite a bit of a shortcut. It already has all the stuff in here that I want and what I'm going to do, I don't need the totals for now, so I'm going to wipe those out. I'll leave all the other metrics and everything there and I'm simply going to change this from d3.csv to d3.json and change the extension here to JSON so it's looking at the right one, I'll save it and then I'll preview it and there you have it. So essentially nothing changed in our code, but the file itself is a different file now, we're looking at the JSON file and it's also being parsed using a different method. So that's the beauty of this is that once we get the data in whether it's CSV, JSON, a text file delimited with some other method, or an API call. When we get it in and it's treated as an object or an array, life becomes very ubiquitous in how we treat the data and how we visualize it, it all seems to be same just give me the data and then I can treat it as an array or an object and work with it from there. So this is the first example, I'm going to leave it here and now let's get into a little bit more complex one. I'm going to save this as working-with-json-2 and here what I want to do is something very similar except this time I want to show the header. So I'm going to look at the monthlySales by category JSON and I'll go ahead and close the original one, I'll close my preview window. If you remember what this looks like, it has this header information. So let's say we want to show that in our viz, well, let's create another function, just like we did here, I'll leave the line one alone because the line one should be fine, function showHeader and in here it's really straightforward d3.select body append, I'm just going to append an h1 a heading tag, text of ds.category + Sales 2013. I could add the region in here, I could add all kinds of other stuff, I just want to illustrate this as a very basic example. I'll give the function call down here showHeader and there's one thing that's different here. If I were to run this right now, I'll go ahead and Save it and Run it just to show you want happens. I don't get my line and why is that? Well, if you recall how this works, this is a nested object, so I have category, which is at the top level and in my file I'm referencing ds.category, correct that makes sense, but now this month end sales, which are used to draw my line are actually nested in an element called monthlySales, so what I need to do now is I actually need to drop down on my buildLine and these guys are failing because month end sales are not at that top level. So, it's a no matter, what I'll do is when I call this function I'll actually pass in monthlySales, so now my ds.monthlysales is essentially the old object that we were working with, it's this bit of data here, it's this array that lives inside of this monthly sales. So now, when I run this again, Save it, Preview, I get my line. So there's a little bit of a lesson there. Because JSON can be nested the things can become a little bit more complex when we're trying to access that data versus the CSV which is totally flat. Now let's take a look at another thing we can do with the data in JSON it has to do with having multiple categories. So just getting a little bit deeper and more complex actually probably more common as to what you'll normally see in the real world.Let me close my Preview window, close the JSON we've been working with, I'm going to Save this as json-3. Now I'm going to take a look at my monthly sales by category multiple and here I have even another level, contents, category, and then monthlySales. If I scroll down you could say I have technology and a furniture category. Similarly, I could have multiple regions, I could have all kinds of dimensional attributes that I can reference in a single document. Now what I'll do, I'm going to use this one, so monthlySalesByCategoryMultiple and because I have multiple in here I want to draw multiple lines. So I'm actually just going to loop through this. So here I can actually say data.contents which is that top level element foreach, which is a foreach loop meaning it's going to call whatever I type next. For every array in that element of contents, function ds pass in my functions here and just to see what we're working with I'm going to hit console.log ds so it'll log into the console so I can see what data we're actually talking about and then I'll do a showHeader and a buildLine. We're actually going to get rid of this global variable because we're actually passing that in now and I don't want to get it confused, similarly I'm going to get rid of all these other variables that are just lingering around. Okay, so we have showHeader and buildLine, these guys need to accept and incoming parameter that is the dataset or ds. So now, when I call them I'm actually going to pass that in. I'm going to make sure the function is correct. So the way this works is it's going to take a foreach, everything in this contents which is everything that's in my data that's being returned from my function call and it's going to load it into this variable called ds. I'm going to log that to the console and then I'm going to run these functions on it. Let's see what we get. I'll Save this, hit preview, tada, now I have multiple lines being drawn for the two different categories inside of my data. This is a great way, an intro to how to build these visualizations very much focusing on a business context, but very practical because now what you'll see are something of spark lines and just like you saw before with the CSVs, we can add the totals to these or even before when we were just looking at building shapes you can add labels to different points or color code these, a lot of things now in your power to make a visualization that actually conveys meaning to your users using D3. Let's go back and recap this module before we move onto the more complex ones of working with data in an API. Summary So we started this module talking about external data sources and the different types there were. You have the flat files, the static files that you're working with that are a local resources on your server. Then we also talked about APIs and the external dynamic data sources that we can reference using AJAX methods GET or POST. Then we took a look at a deep dive in working with CSV data, actually working with that data on our local server and all the different ways to parse that and draw things using it. Then we took a look at JSON data and JSON data is very similar to CSV other than it can be nested and it can have multiple levels in it so working with it can become a little bit more complex, but really is going to be the focal point for the rest of the data we use throughout this course because that's what you'll see most likely on the web as it really has exploded as the most common way to transfer data around because all programming languages can work with it very easily and a lot of programmers like it because it's very easy to read and consume visually. Working with Data Part 2 - Getting Data from Web APIs Introduction Hi, this is Ben Sullins with Pluralsight and welcome to the second part on this module of Working with your Data in D3. In this part of the module, we're going to look at getting data from a web API. We're smack dab in the middle of the course and the real tactical part of actually pulling the data into D3 and doing something with it so we can then visualize it. We're going to start by looking at a web API overview. I'm just going to give you a 50,000 foot view of what web APIs and how you can interact with them and some of the ones that are out there on the web. We'll take a look at calling the web API for data and actually retrieving that and visualizing it. I'll look at decoding data a lot of times when you transfer data over the web it gets encoded and we have to look at how to decode that all using JavaScript and then I'll share some of the public data APIs that are out there for reference. Let's get started. Web API Overview For the Web API Overview, I'm going to go over and just show you a couple examples of this and talk through what they are and then we'll get into how to actually access them. Here I am back in Cloud9 with the example we just left off of working with our data in JSON and I just want to switch over to data.gov real quick. On data.gov there's a lot of data and it's a really interesting rich place to get information for your visualizations to add context. They've recently redone their site and I've think they've done an excellent job in making it simple and easy to find interesting datasets to use. One of the interesting things we're going to do is go into the developers area and this is where we're going to be able to find our APIs and when we click on this, we actually see a couple of different examples here. You have this CKAN API, you have a US Government API and you have all the different examples of those there and what I want to do is click on this API documentation available here and in here, we'll find an API guide. Now remember what we've been talking about before is getting JSON data and there you go it's live on the web. The most common way you're probably going to get data is in that format and if I take a look, there's all these different URLs that you can hit. So before I was typing in the name of a file and that file lives on our server, so it's really easy to access it, just like if we were coding HTML, we put an image somewhere. Here, what I'm going to do, I'm actually going to send a request to this URL and it's going to send me back this big giant JSON blob and as I mentioned before, as you can tell they probably get a little bit more complex the deeper you go. You can take a look at all different kinds of these and all the different datasets that they spit back. So, you can go down it explains here on how to make an API request you essentially run a GET and in D3 we use a command called xhr and what it'll do is it'll actually pullback something that looks like this, something that looks like JSON and we've just learned how to iterate through that as an object referencing the different layers inside of it and most of the documentation you find out there talks about the different callback methods you get success or if you get some results. In this case, there's usually a standard format and here's even an example using Python, not JavaScript, on how to actually make that call. So, when I talk about a web API I'm generally talking about this. It's also known as a rest API. Rest APIs are the ones where you just basically send in a URL request and it'll spit back some result and then you parse that. There are other types of web APIs, but there more focused on applications and actually working with objects in transferring objects back and forth. Here we're simply talking about going and getting data from a URL that is exposed on the web that we have access to. DEMO: Calling a Web API for Data Now let's take a look at actually calling web API for data. The first thing I want to mention though is that there's something we're going to run into very quickly when we do this and we need to be aware of this, it's called a Same Origin Policy. This is something that is built-in to web browsers and built-in to web servers and its long-standing and what it basically says is when you try to access something that's external, something that's not hosted on your own domain or on your own server, you could be injecting or opening up your code for hacks and malicious attacks. So let's take a look at some examples here. If I was at bensullins.com and I was trying to access bensullins.com/data, this would work. It's the same domain, it's just a different page. Similarly, if I was trying to get at bensullins.com/data?page=2, totally works. If I'm trying to pass in a user name and password to bensullins.com/data, still works, same domain, everything's good. Now what if I was trying to hit bensullins.com on port 81, now, it's a different port. If I was trying to https versus http, it's a different protocol and lastly, if I was trying to get data from a completely different site, the obvious answer is no. Now this is what the same origin policy dictates and like I said there, it's a configuration setting in your browser and on the web servers themselves. So it's possible to get around this by changing the config or in the newer way of transferring data on the web, there's other protocols that don't get blocked by this because they have a bit of a more complex way of handling the security. Now I'm not going to dive deep into how web security works here, I just wanted to introduce this as something that you need to be aware of because you're probably going to hit it very, very soon as you start to develop here. Now in this example I'm going to use, we're actually going to go get data from GitHub, but GitHub has an actual API and most things that see label API usually have the built-in protocols necessary to work around the same origin policy. So we shouldn't hit that error and with shouldn't have to worry about this now, but just know as you get into this, there's a lot of other things to consider besides just the code that it takes to actually make something happen. So let's dive in, let's go back to our Cloud9 environment and start coding. Here I am back in the Cloud9 environment and what I'm going to do basically is try to rebuild this exact same chart that we had or these two charts that are being generated by this monthlySales by category multiple JSON file and instead of using it here, I want to pull it directly from GitHub. So let me switch over and show me GitHubs API. Here under developer.github.com/v3 or if you just for it you'll get this page has all of this documentation on how to use GitHub to actually make requests and actually do things and just like I said before, it's basically hitting a URL and parsing the response. Now, fortunately, we're looking for one called Git Data and we're going to pullback what is considered a blob and a blob is essentially, it used to stand for binary large object, essentially now what it is, is a bit of data you want to retrieve and you can see the example here, GET which is a type of request if you remember from our last module when we talked about the different types, a repo which is a repository on GitHub, the owner, the repo name, get blobs and then sha which is a hash value that represents essentially the name of it. So what we can do is I've actually uploaded this already, so let me show you the repository and you can follow this along and I'll leave this up here for you to use and you can try it with your own GitHub repository. So if I go to github.com and I login you can see my account there. I can actually go down to my repositories and I have one here called d3js-resources and in here I have a couple different things. I have an image I have the CSV that we used, and have this JSON file, which is the one we're actually going to use for this module here and if you can remember it's actually the exact same thing that we had in the other one, it's that contents with the categories of furniture and technology and then the monthlySales embedded in there. So what I want to do is I want to actually pull this data live from the web, which means that if this were to change, it would actually change my chart as well. So in order to do that, I need issue that API command and so what I'll type in here, I'm going to go to the beginning of this URL, I'm going to leave the https, I'm going to add api.github.com, this time it's going to be repos/bsullins/name/contents and then the path to the file essentially. So this is another way to get the data back and when I hit this, you'll see some JSON that pops up. Now, remember, just like we had before, we have the header information, name, path, that hashed value, different URLs, and then we have the content. Now the content here is essentially that file. This is the file that we're looking at and this is the data we want. You can see the /n for newline characters, and all those embedded in there and then you see just what looks like a bunch of jibberish a bunch of mixed up hexadecimal characters and then I get a clue here at the bottom that there's actually something that is encoding this. So what I need to do is decode this data before I'm able to use it in my visualization. So let's take a look at how to decode some data now. DEMO: Decoding Data This data is encoding using a64 and then there's other encoding methods, but this is the one that I seem to run into the most, so I'll cover that and then I'll point you to the place where you can find out how to do this on your own if there are other types of encoding methods that you run into. I use the Mozilla Developer Network quite a bit and on here, there's something that is Window.atob and this is one that actually will decode the data, as you can see right here, it decodes the string of data which has been coded using base 64 encoding and then you can do the reverse of this and it would essentially encode the data and transmit over the web. Now if I wanted to just go one step up here to the web API interfaces. I like to point this out because we've been talking APIs as ways to get data. APIs are also just ways of interfacing with other programs and other pieces of software, other libraries. So for example, you can have an API that is specific to your programming language, whether that's C#, or Java, Python, or PHP, and that API can essentially be a library or something you work with inside of your language that then interacts with some other program, some other platform, and this is what is so powerful about these is that they're able to interact and allow your .NET application to talk to my php web application speaking the same language, and so we have these endpoints, these Rest APIs, which return the data like we've seen with our JSON data being returned here and then we have the APIs which are essentially just libraries within our language and a web API here is essentially that is used to talk to a web application using JavaScript, JavaScript being the primary language of the web. So let me go back real quick , to our function that we need to call and this is basically all we need to do here, so when we call our data, we're going to call that API from GitHub, it's going to return the contents, then we're going to decode it and then we're going to pass it into our functions to draw a visualization. Sound easy right? Let's jump right into it. If I go back to our visualization, first I'm going to Save this As working-with-api.html. Here, I'm still going to use .json. I'm actually not going to change this. There is another called xhr, but because the data being returned is JSON, I actually just leave it as JSON and it issues the request to whatever I give it. So if I Copy that URL form GitHub and I come back here and I Paste that in I essentially have this same code as before except now I'm referencing that external file. I'm calling an API to get that data. Now if I wanted to just see that there, the data when it gets logged, it's going to look a lot like the contents that are displayed on this page, all this hexadecimal base 64 encoded data. Well, I need to do something to parse that, so I'm going to have a new variable here. I'm going to call it decodedData and here I'm actually going to call something json.parseWindow atob data.content and so what this is going to do it's going to take that function that we just looked up on the Mozilla Developer Network, and it's going to decode data.content. Well, data we know is what we're getting back and that's the variable that that we're loading things into from our API call and if you recall how JSON works, we're going to hit this array here, this content object and it's going to decode the value of that which is this base 64 encoded string. So let's keep trucking along here. Just for the sake of it, let's say console.log decoded data.contents and for now, I'm just going to comment these guys out and I want to run this and see what I get, make sure that things are working properly. So when I Save that hit Preview, it gives me blank page, as expected. I go to my Inspect Element and then I hit the console and you can see there it is right there. So it first logs the data in this call here, the initial call, just an error check and then it shows me that there and then when I dig into that at the very bottom I've decoded that and when I decoded it I've listed the contents which are two additional objects. I could add to this the first element in here and it's a 0 based index, so just putting a 0 in brackets would give me a first object to my array, Save that, and hit Preview, scroll down to the bottom and you can see it gives me the first one there. If I wanted to similarly do the second one because I know there's two, enter this guy, and there we go, so it prints out the full thing and then it prints out the two arrays that are living inside of my object. Okay, so the last part we need to do now is to actually use this data in our functions that already exist that are drawing these two line charts and here instead of just using data I have to actually use the decoded data one. So I'll use decoded data.contents and I'll loop through those, passing in the results of each into ds it'll log that and then it'll show the header, and then build the line. I'll Save this and hit Preview and there you go and as you can see the logging all works correctly, so I logged the full thing, I logged object, object, which is decoded data, and then as I went through I actually logged each of the objects to the console there, and then I pass it in, and the function again didn't change. If you recall I'm going to switch from CSV to JSON we didn't really have to change much other than we learned about nesting, but here, once the data came back as JSON, it was the exact same functions worked the exact same way. So it became ubiquitous, the data is handled very much in the same manner once we parse it into the correct format that we're used to working with. Public Data APIs Alright, so now let's take a look real quickly at some different APIs that are out there on the web that you can use to enhance your visualizations and really come up with some amazing creative things. We've already looked at data.gov and the different examples we have there. We also have Twitter, under dev.twitter.com and this one's a bit more complex because what we have to actually do are issue credentials and we actually have to log something and then once we do that we can get data out and we can do really fascinating things. I was visiting Twitter just a couple weeks ago and someone there used D3 to build something that showed a world map of every tweet popping up as a bubble bursting and it would focus in for 30 seconds on the region that had the most active tweets an at the very top of the visualization it would pop-up the tweet that was the most popular, the most retweeted, favorite etc, for that 30 second window and it was pulling in streaming data from the Twitter API and visualizing it in real-time using JavaScript. It was a beautiful stunning piece, one that definitely was worth the record books and I think that just illustrates the types of stuff that you can do and how powerful this language really is. The other API I want to talk about is GitHub and we've already looked at GitHub a bit, but I just want to go back and say that this is a really great way for you to upload your own data, you can have a secure account where that data becomes private, and you can access that. I use this in real life situations all the time. My users may have custom data input that they need to embed into the visualizations, let's say a grouping of something and with that grouping, I don't want to have to hardcode that into my visualization and I don't want to have to certainly handle the data transfer, them sending me an Excel file and the popping out my server or anything like that. I can publish something in GitHub, like this blob that you saw, I can give them contributor access, they can edit that, and see the results in real-time on the visualization. So GitHub is another great one and their API is pretty powerful. It can do a lot more than just get data, it can actually interact with the platform, but I use it mostly just for data transfer and that kind of a thing and I'll show you another one real quickly which is kind of fun and as it goes now Bitcoin is all the rage and so coin base, which is sort of the PayPal of the Bitcoin world actually has an API which you can query things and get data out. For example, there's some historical one. You can also interact with it like I mention before, you can actually build something on top of it, but there's some fun stuff here where you can actually pull in historical pricing and those kind of things. There's all different kind of options and stuff you can do here, so you can see the different currencies, see orders. If you were building an application on it this is what you'd actually use to interact with their platform. So coin base is another interesting one that spits out different types of data and different ways to interact with Bitcoin information on web, pretty cool, pretty powerful stuff. Alight let's recap what we've done now and move onto enhancing our visualizations. Summary We started by taking a look at the web API overview. I showed you some examples of that on the web, gave you some context, looked at some of the different options that are out there. Then we dove into actually calling a web API for data using GitHub's API to have data that's stored in a repository accessible inside of our application. We looked at decoding that data and one of the key functions that you'll need to know when you deal with a lot of the data that comes from these APIs, and then we had some fun taking a look at some of the Public Data APIs out there. Enhancing Your Viz Part 1 - Scales and Axis Introduction Hi, this is Ben Sullins and welcome to this module on Enhancing Your Visualizations using D3. This is a two-part module and the first part we're going to talk about scale and axes. We're trucking along here getting towards the second half of the course and this is where we're going to really try to fine tune our visualizations and make them presentable to our end users. First we're going to talk about scaling. What are we talking about when we say scaling and what does that mean in D3? We'll actually go into an example of it. We'll talk about adding axes and referencing those in D3 and how we create them, and then we'll going to through a functional example of actually doing it. Let's get started. Scaling Overview First, let's understand what scaling is in D3 and to do this, I'll pull up a quote by Mike Bostock who's one of the founders of D3 and he says that "Scales are functions that map from an input domain to an output range." So those are the key things here we have an input domain and an output range. Let's take a look at an example. Let's say we have some sales data here and we have Year and Sales in thousands. We'll the input domain is going to be the min and the max values of our sales and that's going to give me something of a number line like this from 130 to 350. Now the output or the range is going to adjust that 130 to 350 into a pixel size that I can use to draw my visualizations. So imagine if as we've been doing far we are setting the width and the height of our SVG and then we're manually dividing and calculating where things should appear within that window, as it relates to the data, but instead what's going to happen, imagine if our data next year was at 5000 or it just exponentially grew, we'd have no way of dynamically adjusting our visualizations so you'd see the data, the data would appear lost, and so domains and ranges or scaling is the way that we can adjust the data values to fit within our expected output range. Let's take a look at an example here. First, we start by creating a scale and in D3 there's a function called scale and then you tell it what type of scale. We'll use linear for this course, but also look at the log or the power and there's also many other ways of creating different types of scales. For most cases that I come across in the real world, I use linear because it's the one that makes the most sense. Sometimes I do the log scales and those are if I'm doing some more statistickal things or my data has a very wide distribution, something that's in the billions, but everything else is in thousands. Once we create the scale, we assign the domain and the range, so if we looked at our previous example with the sales data, our domain, that the min and the max of the input data we're getting, of our sales values, and the output range, how big is our SVG going to be. Well let's say the minimum is going to be 10 pixels and largest one is going to be 100 pixels, so this will give us that scale of what it looks like and if we wanted to use this, we would simply write it out by calling our function and it would output the values there that you can see. Let's do this real basic example and then we'll get into some more tangible ones and actually using the visualizations we've been build this whole time. Let me switch over now and I'm going to go to jsbin.com. DEMO: Scaling Data Here on jsbin.com I have my template that I've been using that references D3, so we need that to make sure things are going to work, I'm actually going to hide the HTML for now because I'm not really interested in that. I'm going to open up the JavaScript window and the console, the output here. So for this to work I'm going to simply do, well I put on the screen there, scale = d3.scale for demonstration purposes I'll line them all up here, linear, domain was 130 to 350 and the range was 10 to 100. Now for our values in here, they need to be in brackets, so that way D3 knows that it's an array meaning I have two elements I have a min and a max. Once we've got that, now we just go ahead and use it. So I'll hit console.log scale 300 and you can see it output there 79.545454, let's just give it another example here console.log scale 270 and we got 67.27, console.log scale 150 18.18. So what you see there is we have data points that are being sent it, so we're sending 300 into our function here, scale, and it's going to apply our scale to it that we've defined using D3 and then it gives us an output number. So remember, 350 is the top of our domain, so 300 is pretty high on that list and we need to map that essentially somewhere in between 10 and 100. So that happens to be 79.54, so close to 80% of my range is where this 300 value maps to using the given domain, so on and so forth. So when we think about how we're going to use this in our visualizations, essentially, we're going to replace what we've been doing before where we change the x and the y values and we divide them and kind of manually put them in there or if we had data that was extremely large and we weren't expecting it, we can dynamically change where those elements show up on the page using scaling. So let's do that now, let's revisit where we've left of within our last module and we have our visualizations coming in from the API a couple line charts, and now let's use the scaling instead of just manually putting in where the x and the y coordinates are going to be. So I'm going to switch over to the Cloud9IDE and reopen the file we had been using and I'm going to Save that as a new file and then we'll start adding our scaling functions. Okay, so here I am back in Cloud9IDE and I've opened up the file we finished off with in the last module, working-with-api.html, that one will be in the demos.zip folder of this module as well, and if you recall what we're doing here, we're going to get some data from GitHub, we're decoding that data, and then we're iterating through it with this foreach loop and we're drawing a header and some visualizations. So if I hit Preview, you'll see what I have here, is I have my two different sales categories and the line charts for each. Notice that the lines are actually running off the page and that's something that we're going to try to adjust for here using scaling, close the Preview, and now what I want to do is essentially where the x and the y are, I want to replace these with scaling functions. Well first, I need to create them, so to make things a little bit easier to read I'm going to hide these two functions and inside of the buildLine function I'm actually going to create new functions to create our scales. So the first one I'm going to call it var xScale and just like we did in the example d3.scale.linear. Now for the domain, it gets a little bit trickier because we're talking about a dynamic domain and not something that is static that we just type in, that would defeat the whole point. So I need an array, so I'll use my brackets and for the first things I'm actually going to say d3.min ds.monthlySales function d return d.month. So what I'm doing here is I'm using this min function, earlier in the course we used this to find the sales total and the sales average, so here we're essentially doing the same thing where we're looking at ds which is our dataset being passed in, the monthlySales node or array, and inside of there we're going to iterate through that looking at the month value to come up with the min val for our domain. Remember, our domain is our input so it's what we're getting from our data. So that side is the min and then the next one is going to be the max. So this is our x scale, our domain is set. We've got the min and the max, now we need to have the output range. In this case, the range is going to be from 0 to the width of our SVG. So we're using the database up above which is also set as the width of the overall window that we're looking into and we're going to use that there to define that. So it's from 0 to 400 in this case and if our SVG were to change size and we could make that dynamic, let's say by looking at the resolution of the screen we're on, then maybe perhaps we could adjust that as well. So that's good for our x scale, don't forget your ;. Now we're going to go ahead and create our y scale, very similarly, d3.scale linear domain now we're talking about the y scale here so we're talking about our sales values, so kind of want to run these from 0 and that's just a data visualizations principle. It's something that we want to use to always have things with a 0 baseline, so that way with the trends and everything are actually representative of the true data values. There are cases when maybe you would also like to use a min here, but in this case, we're going to 0 as our baseline. Then we're going to pull in the max of our sales value. So just like above, d3 max ds monthlySales function, return d.sales. I'll collapse it here so it's a bit more condensed, this one's a bit easier to fit onto a single line. Now I'll add the range and the range in this case is going to be from the height down to 0 and remember that the way that the coordinate system works is its top left not bottom left is 00, so we have to have the height first versus 0 first, otherwise our line would essentially be upside down. Don't forget the brackets to tell it that it's an array and we should be good. So now, we have our x scale and our y scale. Now, down here, when we actually create the x and y coordinates for our line, instead of having the actual calculation by hand that we've been doing here, what we're going to do is we're actually just going to call x scale and pass in the value that we're looking for, x scale d.month. Similarly, we're going to pass in y scale d.sales. So just like before, what it's going to do, we have two functions, x scale and y scale that define what our scale is and the x scale it's from the min to the max and from the y scale it's from 0 to the max. Then as we try to draw our line using the line operator, we're going to pass in the month value to our x scale function and it's going to return essentially somewhere between 0 and w somewhere between 0 and 400, as to where this value falls and then that should give us the actual coordinate that is going to be displayed properly and still be true to the data and representative of what the data's actually showing. Same with the y scale, this time we're looking at sales and it goes from h to 0 and it'll pass that value in and return somewhere between those two values where this data should actually fall to be still true a representative of the actual data, and now that we have everything lined up, we'll go ahead and Save this guy and then Preview, and viola! You can see the lines are still being drawn. They actually moved a little bit to the left and if we want to adjust this, let's say make this 300 Save it, run it again, you can see the lines get a bit more condensed. Similarly, if we wanted to make this 50 high instead of 100, you can see how things get adjusted here to fit within the window that we've created. So that's really important fundamental piece of any visualization that you want to draw because it's most likely that you're going to come across data that is sometimes unexpected and when you do, you want your visualization to be responsive and be able to handle that. Now let's move on and take a look at adding an axes Axis Overview So I'm going to talk about adding an axis, we're talking about the labels that go along the bottom and the edge to give context about what the data is and what we're looking at. So to create an axes in D3, we start by calling a function called axis and you need to reference the SVG first. Usually we put this into a variable, which is also turns into a function that we can call later. Once we do that, we add the scale, which we just finished building and this tells the axes essentially the min and the max values and what that looks like. We give an orientation, in this case for x axes we'll do a bottom. If we wanted to do a vertical axes we'd do left to right, and then we add it to our SVG, pretty straightforward. Now let's take a look and add one to the chart that we've been building. DEMO: Adding Axis to Charts For this demo, what we're going to do is we're going to dive into adding an axes and we're going to build on the viz we've been making that we just built the scales for. So I'm going to jump back over to the Cloud9IDE and I'm going to continue where we left off. Okay, so here I am in the Cloud9IDE and I have my scaling file that we just finished with. I'm going to go ahead and Save this as a new file, and I'm going to call it axis-1.html, click Save, go ahead and close the preview for now, so we can see all of our code here. So what we're going to do is we're going to start with a y axis because it's a simpler one and then after this I'll show you how to add the x axis which has a bit more complexity because it's date values instead of numeric values. First thing though, let's just add some styling here to clean this guy up. If we hit Preview, you'll remember what this looks like, pretty boring, pretty bland, so I'm just going to add a little style tag here and I'm going to actually add something for my axis. So I'm going to call it axis path, axis line, so these are CSS methods and classes that we're going to be referencing. We're going to add a fill for none, stroke black, shape-rendering crispEdges and these are kind of a common set of styles that I'll use on all the axis. Also for the text, let's make the font-family sans-serif, a bit cleaner, easier to read, and for the font-size let's put 10 pixels. Lastly, let's add one here for h1, which is our heading and we'll do a font-family of sans-serif. Okay, save that, Refresh, let's see if we can do font-weight 300. There you go, things are looking a little bit cleaner, a little bit more modern, easier to read. Maybe the last thing that we'll do is we'll add a size here, font-size let's do 16. There you go. So now, things are even easier to read, they're beginning more compact and you can adjust this, this is just for the demonstration purposes so we can see the full text and everything. Now notice how the chart goes off the page there, let's go ahead and adjust that, let's pull the width down to 300. There we go, so we got something a little bit cleaner, easier, to see and now it's time to add our axis. So we already have our scales and we have our functions to build the line and everything like that. Okay, so I'm going to close the preview, now that we've added some style elements and I just need to basically add some functions to generate the axis. I'm going to go ahead and do that right after my scale function because remember we have to actually reference those scale functions. So first, I'm going to do one for the y axis. So I'm going to call it y axis, just like mention d3.svg.axis.scale and this time we're giving it the y scale so it'll send the value to that function up above and we're going to orient this as left. Okay, so we've created our axis, now we need to actually add it to the chart. So down here just below right before path, I'm going to add a new one called axis svg.appendg. G is a group element in an SVG that allows us to put all this stuff in together and it's kind of like a div tag where you can group other elements inside of it and this one is really useful for axis because the axis actually generates a whole lot of stuff when it gets rendered and what we're going to do now is add a new operator call and call is going to add the y axis. Let's add some attributes, here. We're going to add that class, that CSS class that we had above and here's a new for us too, this is So what we have there is essentially, it's going to move this left 20 pixels or whatever our padding is set to, 20 pixels, and then that's what going to allow to show up. In fact, what I'll do for demonstration purposes, I'll comment that out, add this, and we'll see what it looks like. So if I hit Save and then Preview, you're going to see what happens. You see that we know have what appears to be an axis on the left side, which is what we wanted, but you can't really see it and the chart actually butted up right up against it. If I Inspect the Element, we go down and we take a look, we have our SVG, we have a path, which is our line that we're familiar with and then we have something here called g and this is actually those tick marks and you can see what it's doing. It's trying to put them there, but we just can't see them and as I hover over, you'll see it move up and down. So what I need to actually do here is I need to go in and I need to apply a transformation to just move it over to the right, just a hair. I'll Save, Preview, and now you can see it appears. It looks a little funky, but don't worry, we're going to fix that and also notice that the chart now is labeled right on top of it. So let's move the chart off to the right just a hair, so that way we have room to see our actual axis. So in order to manipulate that, what we need to do, we need to go find where the scale gets generated for our x axis and the range for that is 0 to width meaning the chart is going to take up the entire space of our SVG and what we want is we actually want to have it start at the padding and then we're going to do w-padding, so it'll end with the padding on the right side as well. I'll save this and hit Preview and now you can see it moved it over to just where our axis starts. Let's give it a little bit more +5 and there you can see. Now of course at this time you could add a different variable to say axis padding, chart padding, SVG padding, you could have all these different ones, but for the sake of this we're just going to keep it simple here. So next, what I want to do is adjust the y scale because that'll make the axis fit within there, so right now we're again all the way from 0 to the top of the SVG and that's why our labels and things kind of fall off our outside of that window. So I'm just going to do h-padding and we'll just start it at 10 let's say and see how that looks and there you go. So now we have a nice y axis. Now one more thing that will show a little bit later is I could also format these tick marks or tell it how many ticks that I want to show and to do that I would actually go up to where I'm generating the axis, say ticks, and give it number, let's say I just wanted 4. You can see now it adjusts and a bit easier to read. Now that also all of these things can be generated dynamically, so be conscience of the resolution of your screen, the size of your SVG, the amount of padding, all that stuff when your adding these to make things visually compelling and something that makes sense. So the data actually is understandable and it's not something that is just incredibly cluttered, kind of as you saw it originally with things falling off the page and a tick mark for every 10. Now let's take a look at adding an x axis. So I'm going to leave this as is and I'm going to Save this as axis-2, hit Save. Now just like I generated the y axis, I want to generate the x axis, close my preview here, d3.svg.axis scale x scale this time orient, this time we're going to do bottom because it's horizontal and we want the tick marks to show on the bottom and I'll leave that like that for now, and I'm going to rename some of these things because down here we were just using axis and we only had 1, but now that we're actually adding a y axis, I'm going to go ahead and give these different names. So this one I'm going to call y axis and up here, we're actually generating, I'm going to all yAxisGen xAxisGen and similar to how I added it with the y axis, I'm going to add another function call here to add the group for the x axis, var xAxis = svg.appendg.callxAxisGen don't forget to change the other one up here. We're going to give it that same class. So the formatting is still the same. This time the transformation's going to be similar except instead of moving it to the left, we're actually going to move it down. So here I'll say translate, I don't need to move it to the left at all so this is going to start with 0 and then I'm going to actually have the height of the svg - the padding and close that guy out there. Okay, so that should add our x axis, we have our new functions to generate them and we're referencing those properly when we add our group element. Now, let's just see where we are. Let's go ahead and take a Preview and you can see we now have an x axis, it's in the correct location, the y axis is still there and also correct, but look at what's happening here with the x axis. Basically, it's trying to put every month on top of each other and it truly is labeling each one of them. If I highlight them you can kind of see it, but the problem is, is that these are integers and what we want are month names. I want to see Jan, Feb, Mar, Apr etc, the month abbreviation because I know I'm filtered to 2013, it says that in the chart title, so I can put that there and it will be nice and compact and this is why I waited to actually show you the x axis because we're dealing with dates now and the challenge is, is that we actually have to convert these to a date and then add a format so it shows up properly. So there's a little bit more coding and the rabbit hole gets just a little bit deeper here. So let's start this part by going up to the top and creating a function to get the data values. I'm going to call it getDate and this function is going to be purely JavaScript, it's going to accepts some data. I'm going to first convert it to a string, so I'm going to call this strDate = new StringD and so this is going to create a new string object in JavaScript that we can then use to manipulate and parse out this date. If recall what the date values looked like, they were full integers so 2013-01-01 was January 1st of 2013. So what we'll do now is I'll have a couple more variables here, year = strDate.substring. Substring is a function that parses the string from a starting point and then a number of characters. So if we put in 0 and 4, it should give us the first four characters which would be 2013, which would be our year. Now I'm going to do another for month, strDate substring, this one is going to be from 4 and it's going to be 2 characters, so it's going to be that middle chunk the 01 in the example above there. Lastly, we'll add day strDate substring 6,2 that'll give us the last 2. Now there is something funny here about month that when we do this, the month when we try to convert it to a date actually starts at a 0 index instead of 1, so 1 would be February, 0 would be January. So what'll I'll do is actually -1 here. Alright, and the last thing we need to actually return a date. So now we've constructed or broken down the actual value we're receiving, now we need to say return new date, give it the year, month, and day and so what's that going to do is this function we pass it in as we generate our scale and it's going to parse our the year, month, and day and then return a date object and once we have a date object, we can actually use some date formatting to make our labels appear properly on our axis. So now, I'm going to go back down and I'm going to change a little bit the scales that we had because before what we were doing was using a linear scale, now we actually need to do a timescale. So, I'm going to say d3.time.scale. We get rid of the linear part of it, the domain are going to be min and max dates, and in fact, what I'll do is I'll jump up right here and I'll create those. So var.minDate = getDate ds.monthlySales. The first array and month is the attribute I want. Okay, so that should give my min date, now similarly, I'm going to do a max date getDate ds.monthlySales ds.monthlySales.length - 1, I'm going to close my preview so you can see that. So the data here we know is sorted properly. If the data wasn't sorted, we might need to do that first and there's a lot of array functions we can use. In this case though, I know the very first value I'm getting is January and the very last one I'm getting is December because I know the data is sorted properly. Okay, so once I've got that, I like to log these to the console just to make sure that I can see them (typing). So I'm going to log those there, let me go ahead and comment these guys out too, so I don't want to see those. Alright cool, so I have my getDate function, I'm finding the min and the max date, I'm changing my scale now. Instead of using these values that I was before, I'm going to go ahead and just use minDate and maxDate. I can kind of shrink this up, a little bit easier to read there. Get rid of this nice function, which is one that actually rounds things. Almost there, so we've got our scale, which is using the timescale now. We've got our function to get our dates, we're parsing those out. Let's go ahead and Save this, hit Preview see what we get now. Getting a little better, we're almost there, as you can see, I have the actual month names coming in which is great. If I hit Inspect Element and go to the console, take a look, you can see that the minDate is Tuesday January 1 and the maxDate is December 1, so that seems to be correct, that's what we wanted. So everything is being processed correctly, now we're just losing our formatting and we're missing the mark on this. Also, our lines disappear and that could be simply because of the path and what it generated. So let's format this a little bit and where we generate our x axis, right here, this xAxisGen at the end of this, I'm actually going to do something called tickFormat, which affects how that looks and then inside of here I'm going to use the D3 time function, d3.timeFormat and the one we want for this example is %b. So what that should do is that should give us a format that is the shortened month name. Let me go ahead and hit Save and now take a look and see what we got. Okay, so the labels are there, but now our lines disappeared, so let's take a look at see what happened. We have our SVG and we have our path. We have the two different axis here, but notice that the path now is wildly different. Look at the numbers, the numbers are way out of scale of what we have been used to seeing and that's because the scale we've been using isn't set based on those numerical values, it's based on the date values, which are treated a bit differently, so we need to work on that and fix that and we can do this pretty simply in the line, where we actually generate the x value. Instead of just calling month, which now is going to be something different, we're actually going to say getDate and send month end. So we're going to use that same function we used before, we'll save it, hit Preview and there we go. So congratulations, you've now been able to pull data in from an API, parse it using JSON objects, draw a line based on this, iterate through the JSON to generate multiple charts, scale that to the size of our chart. As you noticed if I were to add or change this to 350, I'll hit Save, you'll see that it going to a little bit bigger there, and also add some axis labels. Notice that the 80 disappeared, that's because when it does this it tries to be smart about how it labels them and as things got shrunken down, it actually adjusted it appropriately. Summary So in this module, we took a look at scaling what that is and how that works with the domain and range. We've actually done some examples of scaling, that way the charts can, no matter what the data is, fit with inside of the window that we're trying to draw. We've looked at adding an axis and what that is and we've looked at the couple different options and examples of converting a date value and then me showing that using some formatting, as well as adjusting the number of ticks that show on the different labels. I hope you've been having fun so far, now it's going to get even better as we're going to look at ways to make things interactive and fun for our users. Enhancing Your Viz Part 2 - Interactivity Introduction Hi this is Ben Sullins with Pluralsight and welcome to the second part of this module on Enhancing Your Visualizations using D3 on interactivity. We're getting towards the end of our middle section here on Enhancing your visualization a and we're going to start by looking at adding a filter. We'll look at how to animate things, animations usually detract from visualizations, but in some cases when we're adding a filter or changing the data it's really important to make that smooth and feel natural to our uses so that way the overall experience is positive and lastly we'll look at adding some tooltips, ways that users can hover over things and see some more detials. Let's get started. DEMO: Adding a Filter In this section we're going to take a look at adding a filter to our viz and we're going to start by actually pulling up one of the old ones we've created in an earlier module where we created a scatter plot with different labels. So let me open up that file now, it's called scatterplot-3.js and I open that up here and you can see the code. It's pretty simple. We have a dataset that was hand drawn and then we have our function to generate the color for the KPIs and then we have this labels here that has a bit of a logic set that we can actually change what that is down here on the bottom when we actually adjust this. So that's what I want to do. I want to show how we can have a drop down and we can use that as a filter to change what we seen in terms of the labels. So I'm going to Copy this code and I'm going to Paste it into a new document in my Cloud9IDE and here when I Paste that in I'm going to have to add some other header information. Let me just open up one of the other ones like our HTML template and actually that's what I'll do. I'll just Paste this right into the script section of this and I'll go ahead and save this as a new file called filter-1.html. Okay, now that I've got that, let's take a look here and just preview this guy, take a look and see. As you see there, we have our min and our max labels and if recall I can change this to all, Save it, hit Preview and now it will show me all. So I have a way to change that, a function that works interactively and what I want to do now is give my users a way to change that on their own. So first let me go up to the top here, right after the body tag, so before our JavaScript gets rendered and I'm actually going to add something that will be that drop down. So I'll do a new paragraph tag. Inside of here I'm going to have something that says showLabels and them I'm going to have a HTML select object. I'll give it an ID, labelOption that way I reference it in a selector later, and we'll give it some options here. First one will be all next one will be Minute/Max, and the last one will be none. Let me Save the page now and refresh, and you can see I have that here. So I now have a drop down where the users can change something, but nothing happens. Well, what we need to do in D3 is add what's known as an event listener and what this is, is basically something that waits for an event to occur and we can define what that is and then once that event occurs it does something else and so essentially that's what we want to do. We want to wait for the user to change the value in our select drop down and when they do that, I want to change what label option we're using. So let's take a look here and at just the very bottom, after I've already rendered everything, I'm going to go ahead and add a new event listener by saying d3.select and I'm going to select my select object, which is the HTML one we just created, and I'm going to use a new operator here called on. So for this I'm going to say onChange and there's a callback here just like the rest of our things where we can actually get some data and we can have the data or the index, just like we can in all the other ones, and here I'm going to say sale as in selection = d3.select and I'm going to reference it as label option, which is the ID of our select object. We're going to chose the node and this is going to give us the selected option. So if they were to choose all, none, or min max, it'll give us which one they've actually chosen. After we've done that, now we have to do something to update it. So I'm going to add svg.selectAll text, just like we would if we were drawing something, but I'm not going to include the Enter or append or any of that, I actually want to change one of the values of what's already existing there, so I'm only going to add those operators that I want to change when this event occurs. So in this case, what I'm going to do is I'm going to change the data that's being used, monthlySales text function return showMinMax just like above, monthlySales Sales d.sales and now instead of hand coding the option in there, I'm actually going to add my selection and what that'll do is that'll actual make it dynamic. So as the user changes it, it will update this appropriately. Alright, so now once we have that, let's Preview it and see what we get, Save, Preview, we have all Min/Max and none. So what's happening is every time we're doing this, every time I change one of these values in my select object, it's triggering this event because we have an event listener that's waiting for a change on my select. Now if I had a dozen of these and a really complex dashboard, probably be better to reference them by ID or class if I had many of them and we'll show another example here, but this is the most basic way you can add some kind of interactivity to your dashboard. Now just so you're clear, there is something called a filter as well in D3 that is a bit different than what I'm talking about here. When I say filter, I'm talking about a way for the user to interact with the data by providing input. There is something called filter where I could say, only show data in my dataset that is above a certain mark. Similar to how I have this color coded KPI action, so not to get confused, and if you look up the API reference you may find that as well, so talking about different things here and now let's get into something that actually does a filter by allowing us to actually sort the dates. So here I have the access file that we created in the last module, where we left off and we were drawing our axis and showing them on our line charts with the different technology and furniture sales and this is what I'm going to continue with here and we're going to just continue developing on this throughout the rest of this module and probably all the way through our final project. So I'm just going to highlight and Copy all this stuff. I'm going to go back to the Cloud9IDE and Paste it in there where we can see the changed in real-time. Here I am in the Cloud9IDE, I'm going to get a New File here, Paste this guy in. I'm going to go ahead and Save this as filter-2.html and I'm going to go ahead and hide the workspace files okay. So just to remember what this looks like, let's hit a Preview and you can see on top we have our furniture sales and below we have our technology scale. Both scale and have axis and labeled correctly and what I want to do is I want the user to be able to zoom in here, so I want them to be able to see just last 6 months, or just last quarter, or the entire year, so that way that line chart can be modified on the fly by the user. So similarly, just after our body tag here, let's go ahead and add the control. Start with the paragraph tag choose date Range, I'm going to use a select option again. I'm going to call this one date-option. I'm going to add some option values and what I want to do is I want to use this as the number of months to show, so I'm going to label it that something that makes sense to the user, but the values are actually going to be something that I can use in a calculation. So I have Last Year, or I have last 6/months or I have Last Quarter. Okay, so a pretty basic thing. Go ahead and Save this, Preview it, you can see now I have my drop down with my filter, and it's not anything yet, so we need to go down and add some more stuff. Now this one is quite interesting because we're talking about looping through and building lines, and lines remember aren't as simple as bars or circles, where there just SVG elements, this actually generates that path. If we right click on our SVG here and we say Inspect Element and drill down, you'll see it generates this path with these coordinates and that's how this thing gets generated. So what we need to do actually is regenerate that path. In order to regenerate that, the way I propose to do it is to create a separate function that is updateLine instead of buildLine, and it's essentially very similar except now we're going to just change what's already been drawn, instead of appending things and redrawing them. So first, what I'm going to actually do is just Copy this entire function, this buildLine and I'm going to Paste it right below, this one I'm going to call it updateLine. It's going to work the same with the Min and the Max dates. It's going to also create the scales, I'll remove my logging real quick. It's going to create the scales in the same way and it's going to generate the axis in the same way. Let me go ahead and make this a little bit more readable because I'm going to add some stuff to this is second, but for now, we'll just leave it like this. The line function itself stays the same, the one that actually generates the line, but now when we get into actually adding things to the HTML page, what we're going to do is change it up a bit. So, before I do this, you'll notice that on the pending SVG, and what I would suggest is we're just going to change this to select, so we'll grab the SVG and then we'll do something to it. Well there's a problem here that the SVGs , as well as the axis, as well as the path, they're not identified well meaning if we were to have 20 categories and have 20 different charts generated out of our JSON, which is coming out of an API, so it could change without us knowing. That could make this not work correctly because we wouldn't know which SVG, it would in fact just grab the first one and we want to identify specific ones so that way we are updating them correctly. So, to in order to make these identifiable, I need to go modify how the line is built. So let me scroll back up to where that is and here instead of adding an SVG, I'm actually going to go and I'm going to use an identifier. So to the end of this here, as another attribute, inside of my list, I'm going to add id and I'm going to call svg- and I'm going to add the category. So now, when this SVG gets generated, I'll actually have an id value that I can reference later on. For my axis, and this is common thing we'll do, is we'll actually use different classes for the x and the y. Often there's things that they share, but them sometimes there's things that are pretty different about them. So we'll go ahead and we'll add x and y and then for path, we're actually going to add another class and the class here is going to path-ds.category. So were also using a different class for the path itself and all these things just make it easier to work with it comes to updating them. So one thing that's going to be affected by this is our CSS. So we no longer can say just .axis, we actually have to explicitly call these out .x -axis path.y-axis path, x-axis line .y-axis line, so that way everything is formatted the same, but we have the ability to identify them and treat them differently, yAxisText. Okay, so now I've got to remember to make sure that our CSS selected properly, so I have the .yAxis not just the y axis. Go ahead and Save this and hit Preview. Okay, and everything still looks good, still looks normal, nothing happens yet when we change it, but we've now added a way to identify these. If I go in and do an Inspect Element here, I'll actually take a look path furniture svg-furniture, different classes, svg-technology. If I drill down here you'll see path-technology. So what we have now are more easily identifiable ways of finding this data. So let's go ahead and close these guys and get back and let's finish our update function now that we have things in a way that we can actually work with. So here, where I would be appending the SVG, now I'm actually just going to select the SVG, but I also need to add my identifier, well I have the # the - and I'll add ds.category. So that way I'm picking up the right one and I'm not going to set any of the attributes here. I just need to select that. Then similarly, I'm not going to append anything for the y axis, I'm going to selectAll and I'm going to go g.y-axis, which is how this selection would work and I'm going to call the access generator and I'm going to get rid of all the attributes, I don't need to set any of those. Do the same thing here for the x axis, and again, we don't need any of the attributes to be set. Okay, so now we've selected everything in our update method, and let's go ahead and apply some stuff here, get rid of the attributes that we're setting on the path, and of course, make sure that we're selecting the right path. So, we didn't have to write a lot of code there, but there still is a lot going on and basically what's going to happen is we're going to filter the dataset based on what the user selects, then we're going to pass it to the updateLine and it's going to go through and do all the same line type stuff that the build one does, instead of now adding it, it's going to modify what's already on the page. Just to make sure we're still good, I'm going to go ahead and Save this guy and hit Preview, there we go. Okay, so everything's still generated, nothing's affected negatively by this. I do that often because when the code gets longer like this, it's easy to miss a ; here or a parentheses there, and it's just nice to make sure that everything is still working on the viz and you don't end up with 50 errors, you can kind of pick them up as you go. Now I'm going to jump down to where we're actually pulling in the data in the d3.json call, and essentially add another call here to the updateLine if the select object changes it all. At the end our loop here, which generates the images to start, I'm going to add my event listener. So it's d3.select, we're looking at the select object, we could identify it using the identifier if we'd like, but there's only one on the page, so now worries. On change, we have a function call, give it the d and the i just like normal, don't forget my ;. We're going to get our selection just like we did in the previous example, so d3.select. We're going to do #date option. That's the id of our select HTML element and now we're going to get the node which is the selected one and the value of that node. So now we've got the selection, we'll go ahead and decode our data once again. I'm going to Copy that from above and then similarly, we're going to do a loop here. I'm just going to Copy this whole thing. Get rid of the console logging and change this from a function content to just ds. We don't need to redraw the headers, they're already there and now we're going to update the line instead of build the line. Now, the one thing we're missing is actually filtering, so I'm actually going to say ds.monthlySales which is the array that has the months in them, and I'm going to call a function called splice. So splice is a JavaScript array function that lets me chop up the array and only returns selection that I want and it's going to start by saying, where do you want to start, and we're going to say 0 and how many elements do you want to return? Well, I want to return monthlySales.length - the selection. So if I were to select last year, I'm going to have 12 for the length - 12 is going to 0, so how many are you going to remove, 0. If I had 6 it would cut off 6, if I had 3 it would actually cut off 12 - 3, so it would cut off 9 just leaving the last 3. So let's take a look at this now. So let's go ahead and Save this and Run it and see how it goes. We've got our filter here. Looks like it's not updating the line. So, I want you to understand about how to troubleshoot this, so what's happening here. The axis are being updated and it looks like they're being updated correctly, however, the line chart isn't being updated at all. So what does that mean? That means that somewhere along the line the selection that does this isn't effecting the line, so that would mean that I need to go look at our update function and found out where it picks us up to make sure that it's doing that properly. So if I scroll back up, I can see my append function here instead of a selectAll and then you may notice that we identified the paths based on class, so I need to put my . on their first to identify the path correctly. Okay, now let's try it and hit Preview, there we go. So now what's happening is the event listener is waiting for that select object to change. As soon as it changes, it's going to basically slice up the array and update the line using the function that we created. This is a common way that we can use to modify our displays. Now there's other ways to do it, this happens to be one that I like. It renders fairly quickly and it makes a lot of sense to the end user. Now the one thing that's odd, is if you notice when I choose last quarter, all these tick marks in here, because they're all the same. Well, what I need to do in order to make those correct, is that I should actually change the format or just set the number of ticks. So up here when I update the format, I'm still using that %b which is the month abbreviation, so I could create another function and look at what's being passed in and change that appropriately, but in this case, I think I'll just stay consistent and I want to change a number of ticks and I'm actually going to use a function here ds.monthlySales.length-1. So it'll basically give me the number of ticks, based on the dataset size subtracting 1 because it's 0 based and what that'll do, now, when I rerun it, 6 months and 3 months. So it'll actually identify the number of tics based on the size of the data so it's dynamic. Right now, we're looking at a single year, but had this data been much larger, we might want to do that as well, so try to make things as interactive or dynamic as possible. So as the data changes, your viz can anticipate those changes and adjust accordingly. Next, we're going to take a look at adding a transition. DEMO: Animating Transitions In this section we're going to look at animations and the transitions between things and how the viz may change and what we can do to make that a little bit smoother for our users eyes. I'm going to switch back to the viz we've been working on, the one we just finished adding the filter too, and in here, we have our transition that goes for changing the axis and changing the line. What I'm going to do now is just make that a little bit smoother for our users. So on the part of the update function that actually changes it, where we do the selectAll path and we change the dataset by calling the line function. I'm actually just going to add something in here right before that, that's called transition. If I go ahead and Save this, Preview it, try it again, look at that, it's a bit more animated. See how it's not just switch instantly between it, but it's actually drawing it slowly. Well that's what a transition is, it's a way to change state from one path to the other and draw that in a way that seems that somewhat natural to the eyes. Now, there's a lot of different ways you could do this. We can also add something here called duration and if we wanted to make it let's say really slow, it's in milliseconds, so we'll add 5000, I'll save that hit Preview. Now, it should just take it's sweet time getting over there and you can see that, and the third thing that we might want to consider here is the type of transition. What we're doing now is called linear and so we add an operator called ease and this controls the quality of the transition. So now, when I change it, it should take 5 seconds and be this slow linear progression to get there. Some other fun ones are elastic and I'll change this to 3 seconds, it's a little bit less painful to watch. I'll Save that Preview it, now this one is kind of spongy, kind of bouncy, so it jumps into position and then bounces back and if we go back the other way, you can see it there. Another one to explore is circle. Circle is almost imagine if the transition were on a water wheel that doesn't move linearly like as it gets heavier on one side it moves faster, you can just see what that looks like here, slow and then it speeds up toward the end and just lands right in position. There you go. You can see how it redraws history and the last one is a bounce. You can probably take a guess at what this one looks like, kind of bounces off the endpoint and ends up in the right position. So these are just some fun ways that we can animate things. Here I'm doing it on a change even for a lot of cases I may do this on load, so when the page first loads, things get animated from the bottom or the transition where it comes from and the type. All of this is just other ways to effect how the presentation is perceived in the overall experience for your end user. DEMO: Adding Tooltips Now let's take a look at adding tooltips. Tooltips are pretty familiar to most people, it's when you hover over something you get a little box that pops up and it shows it maybe some more detail about your visualization. There's a lot of different ways to do this in D3, I'll show you a couple of the ones that I tend to like. Here, I have the JavaScript from a file called bar-2.js, this is from the second module on basic charting and I'm going to use this as my first example on tooltips, bars are a little bit easier to hover over than little points on a line, so I'll probably use that for the rest of these and then what we'll do is we'll look at some other options as far as how to enhance the viz we've been working on. So I'm actually just going to jump over from to jsbin.com, and Paste this in, and we'll see what we can do. So here in jsbin I have my template which references D3, that's important. Once I've got that out of the way I can either keep or leave all that stuff, and I'm going to go ahead and hide the HTML window, put the JavaScript one on, Paste this in and you see that I get my bar chart with all the labels nicely put on there. So for this, for right now, I'm actually going to get rid of those labels, just go ahead and delete that piece of the text. I want to use the native built-in tooltips that come with your web browser that are when you hover over something a little yellow box that it shows. So it's pretty simple here to add this. I just basically, need to add something else to the rectangle objects so after my last letter of attributes there I'm going to hit .append a title, which is an HTML element, and for the text of that title, I'm going to have a function and I'll just return the data, whatever it may be and I have my viz there. Notice I can't see anything yet, but if I were to hover my mouse over it, you can see the number pops up in that little yellow box. So this is the easiest, most basic way. It's really simple to add this to any of your visualizations. The one downside or the main reason why not a lot of people do it this is because you can't control much of it, you can't add other links in there, you can't add bold or heading, or any of the other stuff, it also take a second for it to show up, it's not instantaneous and so you might want to use a different method that you have more control over or ways you can make the tooltip more informative, you can even put links or other information in there and also something that you can control when it happens. So right now, it's based on the browser and I would rather have full control over that using an event listener like we were just looking at before. So let's take a look at another example here. I'm going to go ahead and Paste in this code again, so I have the full thing with the labels and what I want are essentially are those labels to appear when I hover over the bar. So that way they're not all displayed initially, but I can actually control when they show up. So like we did before, we're going to add an event listener. In this case, we're going to add it directly to the rectangle objects, so I'm going to delete that ;, I'm going to add a new one called on mouseover we're going to have a function here and in this function we essentially want to add this text. So I'm going to go ahead and Copy or Cut that out of there, Paste that into here, and we're almost there. We don't need to actually use the dataset or the Enter function because it's already using all that as a part of the rectangle object, so all the things that I would normally do, this dataset, Enter, and append, I can get rid of that because all were doing here is actually appending something, so I don't even need to do selectAll, I'm just going to say svg.append. The text, I'm already in a function that's already receiving the data, so I don't need to call it again, I'll just use a d there. For the attributes , I'm going to change the x and the y and it's because I need to actually see where the bar is generated and the width of the bar is dynamic, so however many bars there are this position may actually change. I'll go ahead and leave the font-family and all that. So for me to actually get the correct x and y, what I want to do, I'm just going to wipe out what's there now and now you can actually reference the parent object here by saying d3.select this .attr x and what that'll do is that'll actually get x of the rectangle which is good, but where the rectangle is isn't exactly where I want to be because I need to be right in the middle of it and by the way this is a float so I actually have to tell JavaScript here to parse the float, so there's this function called parse float that does that for us. Now I'm going to add d3.select this attr width, so I'm getting the width of the bar now /2, so I'm taking the position of it, and then the width of it, and I'm plopping this guy right in the middle, just simply by /2 and again, this will be a float, so I ought to do a parseFloat on this guy so it's rendered correctly. Okay, so I've got my x value and it's a bit easier for the y, because the y value I just want it to be at the top and then subtract the size of it, so parseFloat, just like we were doing, d3.select this, get the y attribute and I'm actually going to do a +12. I have the size of it as 12, so I'm going to drop that down my 12 pixels, and we're adding because if you recall, the coordinate system is upside down to what you're used to. Okay, so let me give this an id as well. As you get more into D3 or JavaScript in general, it might be familiar to you to id and class everything so that way you can identify it with selectors. Okay, so now when I hover over this guy, you can see that actually pops up the number, and there they go, and in fact, as I keep going, you see they get a little blurry because it keeps popping them over. We have an on mouseover, but we don't have anything to wipe it out. We don't have anything to remove that from our SVG. So let me just add that real quick, it's pretty simple. I'm going to add a second event listener and on this second event listener it's going to be mouseout. I'll have a function call, even though I don't really need it, and here what I'm going to do is say d3.select tooltip.remove. So remove is an operator that essentially does just what you think it will. It'll remove the SVG element that I've just selected, and now when I hover it over it can see that as I go here, it appears and then it disappears. It looks like I have a warning because I need my ;. Of course, so there you go. I think this is a pretty good way. Now, if we're talking about lines, it becomes a little bit more complex because we're talking about a path and it's not a very linear x and y type calculation, but you can see here that this is pretty good. Now, a lot of people will also change the color of the bar as you highlight it, I tend to think that's not necessary. I think that the color should be more meaningful and we should use color to identify things or inject more qualitative properties of our visualization and less of a decoration. Okay, so let's take a look at another example here. I'm going to pick up where we left off with the transition-1.html file. This is the one where we can change the filter on top and see it update using an animation. I'm going to Copy this and Paste into a new file in my Cloud9IDE and I'll go ahead and Save this as tooltip-3.html. Remember what we're working with, just hit Preview, take a look. If you recall I can change this here and see it update. Alright, so what I want to do is I want to be able hover over the points on here and actually see a little box pop-up with the number, that way I can actually get the quantitative part of this graph as well. So I'm going to close the Preview for now and I'm going to start with some CSS and this is the way where I like to do it, where I get to style something and then adjust where it displays instead of generating tons of elements on my page. So, the thing I'm going to add is div.tooltip and the position is going to be absolute. I'll give it a max width of let's say 200 pixels, the height can be auto, padding let's say 5 pixels, let's give it a background color, let's make it white. Now let's give it some nice rounded edges, order radius, say 5 pixels, and let's give it a box shadow, and we'll just do 4, 4, 10, and we'll give it the rgba code 0, 0, 0, 0.5, so it would make it somewhat translucent. Alright, now it's very important that when our mouse is on top of this box that it doesn't actually take over the mouse events because if we hover off of our tool tip, it needs to recognize that, so there's a very important CSS property here that we need to add and that's pointer events and we're going to say none. This is VERY IMPORTANT. Next, we'll just set a basic font-family of sans-serif and we'll go ahead and do a font-size of 12, something easy to read. So essentially this is what our tool table is going to be styled like. So now let's go down and we're actually going to add some stuff to our buildLine event. Down here, I start to actually generate my scales and everything like that. So what I want to do is actually add the tooltip and this is just basically a div that we're going to move around the page based on where the user is hovering. So we're going to say d3.select body append div. We'll give it that class of tooltip that we just setup and for now, the style is going to be opacity 0, so it'll be invisible. So that way when we look at again, you won't see anything there. You Preview this, you notice we don't see any div tags or something floating around there. So I'll close that. Now just below this path append where we actually draw the line, I'm going to append also some circles. If you recall we build scatter plots earlier, I'm going to call this dots and what these are going to do is there going to actually all the user to have something easier to land on. Sometimes with a line it gets really hard because it's so skinny and so I like to actually just highlight these points to the user. So here I'm going to say, svg.selectAll circle and this is very much just like building a scatter plot, we'll give it our data, ds.monthlySales, we'll hit Enter, we'll now append the circle of any of those that we're missing, give it some attributes here. We'll have our function of cx. It's going to basically, match our scale from above, so I'm going to go ahead and Copy the function we use for the x val of the line and I'm just going to Paste that in there , for the cy value, similarly we're going to use what we have for the line function here, which does our scaling and converts it to the date and does all that jazz that we needed to. Next, we'll give it a radius, let's try 4, we'll give it a fill of that dark grey, and for future reference, I'm just going to give this a class of circle-ds.category. Okay, so now we've got some circles there, we've got some attributes, let's go ahead and hit Save and Preview this and you can see there we have dots on our line chart, nothing too obtrusive you know I could adjust the radius if I wanted it to a little bit skinner. Something like that, or I could make it kind of ridiculous and make it huge, but in this case I think 4 is good. I want to give the user something to land their mouse on that isn't too obtrusive in terms of the trend. You can still see where the trend goes. Okay, so now, that works, but we need to have our event, we need to have the event listener and so right here, not in a separate place, I'm going to say on mouseover function and we'll user our data again. Our div essentially now, so our tooltip and we're going to give it a transition. We'll give it a duration of maybe 500, we'll give it a style now of a opacity, let's say .85. Then we have to set what's in the tooltip, tooltip.html and let's put something in here that's nice and easy to read, so I'll use some HTML code, some strong sales, give it a $ d.sales, notice I don't have to reference the function again because it's already being called on the event there. Let me close the preview window and I'll say sales k because it's in thousands, and then I'll end my HTML tag there. Now we need to position this because right now it won't appear where we want it to on top of there, so I'm going to use the left style and when we reference something inside of an even listener, we actually get some additional properties. One is pageX which is where the mouse is right now and so we're going to add that to the style property, that way the tooltip shows up exactly where we want it to. Similarly here with the top, we're going to say d3.event.pageY and let's subtract 28 from it because we want to give it nice room to breathe so the things doesn't overlay exactly where the users mouse is and remember because we're going upside down - goes up because 0's on top add that so we've got our tooltip that's going to come to life with this opacity and we're going to set the HTML value to something that the user wants to see. Let's give this a go, Save that, hit Preview, and you can see it there, and as I hover around, it kind of pops up and changes. Now I notice that the other ones are still living there, so it's still actually there, so that's because we need to actually add a mouseout event, that way we can clean-up our view for our users. So I'll do very similar type stuff, do a function. This time I'm going to say tooltip.transition duration, let's make this one a little faster.style opacity 0 again. Save that guy, go ahead and Preview it. Now when I hover over, you'll see it shows up and then it smoothly disappears. Very good user experience here and again we can control exactly where this shows up sometimes it may cover up some of the other line, which is another reason why we made it translucent there so you can still kind of see through it. So this is probably the most commonly used way I've seen for building tooltips because you have full control over it. I could put links in here, you can make things bold, I have full HTML, I could size it, I could anything I want and it has a very good user experience, I have a lot of control over how it appears and how it shows up and goes away. So the one thing that I haven't done and this would be your homework is to be able to now update this. So notice this one was made for these transitions, so if I were to change this, oops, my dots didn't go anywhere. The tooltip still actually work and everything else, but notice that the dots aren't moving along with it. I'll post the finished, the completed file that has this working in the demos.zip, but don't cheat, try to figure this out, you're going to basically have to add something like this to the update function. So give that a go. Alight, and let's go recap this module and then move onto mapping. Summary So in the second part of this module on Enhancing Your Visualizations, we looked at adding filters. There's different ways to add filters a common one I like to do is to give my users a drop down and then have all the logic built-in to my code to handle what that means on the frontend. We looked at animating transitions and different options we have for transitions, just another way to make the overall experience quite good for our users, and lastly we took some fun examples of adding tooltips, as well as I left you off with some homework there to try to figure out how to update the tooltips based on the results of a transition. Mapping Introduction Hi. This is Ben Sullins with Pluralsight, and welcome to this module on Mapping using D3.js. We're heading in the home stretch here, and we're getting into some really fun stuff where we can draw maps using all kinds of interesting new stuff built into D3. We'll take a look at the first part of that, which is called GeoJSON. Then we'll get into some demos actually drawing maps, show you how to add color to the map or what's also know as a choropleth, and we'll talk about adding points as in cities on states. Let's get started. GeoJSON GeoJSON is just like regular JSON when you think of it except it has a very specific structure that we can use inside of JavaScript to actually draw shapes or draw maps generally, geographic objects. Taking a look at it, it looks very similar to JSON with all the curly braces or nested arrays, and we have some very key elements here. We have a FeatureCollection on top, then we have a Feature, and then we have inside of that feature geometry, type, and coordinates. So, the Feature is often generally the object that you're trying to draw. It could be a state, it could be a country, it could be a zip code, whatever. The type defines what type of geographic role it will server, in this case a Polygon. That means we're actually going to draw a shape. We're not going to like put a point at a latitude/longitude coordinate. And then we have those coordinates there, which is an array of numbers that is what actually the GeoJSON specification is mostly used for, and that is you pass that into a path operator inside of D3 and it'll generate your shape for you. Now, I'm going to show you how to use this and how to generate it yourself so that way when you're thinking about you know that sustainability project that you're working on and you need to draw a custom map about where your bus stops are and how many people are on there you can use this and find out your own ways to collect and create your own maps somewhat becoming your own cartographer here. Now, the output of this is a path, and we've seen paths a lot with lines, and this one looks very similar. It has these coordinates in it. That's this incredibly long string of numbers, and thankfully that's why we're using D3 here because we don't want to have to generate that manually. DEMO: Drawing Maps Alright, now let's dive in and actually look at drawing some maps. We're going to start here on the census.gov website, and I have this link in the reference slide of the presentation here. So, this is where we can start if we need to get data about shapes and how to draw maps. You can see here we have downloadable files, and the cartographer's of the world have been using these things called shape files for years now. GeoJSON is a relatively new thing with the advent of mapping using JavaScript. And so when you look at this there are some things to consider. One is, you know, what you want to download. We have State, Urban Areas, Divisions, Metropolitan Areas, all this different stuff. And then you have the different size or the different zoom factor, 500K, 5 million, 20 million, and those are going to change from small to big how detailed the map may be. So, if I wanted to download this file here, I'd click on State, and you can see here these are the state boundaries, and you can see that rr is the resolution. And with this I went ahead and downloaded this first one, this 500K, which is the smallest resolution, but still works plenty for our demonstration purposes here. You can see I can download older ones as well. Once I download that, I'm actually going to go, and I need to convert it to an actual GeoJSON because what it gives me are these shape files and KLM files and formats that we can't work with in D3. I switch over to this website, which is converter.mygeodata.eu/vector, and again this link is in the reference module, and here's where I can actually upload my ZIP file. I just hit it there, and then I click through a couple steps, really just next, next, next, and then I have something that is a GeoJSON file based on the data I uploaded. Now, this is a really, really powerful easy way to do this. There are also other ways. You can install a command line tool if you're more of a command line type person, and in fact on d3js.org if you look for the map references there are instructions on how to do it there manually yourself as well. Thankfully for you I've already done that, and there are actually already a bunch of these out there that have been done for the basic type things, the US states, country, borders, etc. But if you're in a different country or if you're looking for something custom, then you might want to be able to do this, and this is the simplest way I've found to do it. Now I'm going to jump back over to our Cloud9 IDE where I've already uploaded the GeoJSON files, and I'm going to start with a new file here. I picked up my html-template, and you can see I have my us.json, this is my actual JSON file, and I just want to draw a basic map here. So, I'm going to go ahead and get rid of my d3 code there. We're actually going to be using a different version of D3. It's the new one, which has some enhanced capabilities here, so it's d3.vs.js. And this is also in the demos.zip, so I uploaded that. And so in here I'm going to start basically how I normally would. I'll have something for the width, give it say 500; something for the height, maybe 300 to start. Now, it's pretty simple. We need to have a path just as I mentioned before, and for this there's something built in called in called geo.path. So just like before we had that line generator, which essentially draws a path for us, it's basically a shortcut for drawing the path, well here we have one that is a shortcut for drawing the geographic shapes. We need to set up a svg just like we normally would, d3.select("body").append("svg"), give it our attributes width:w, height:h. Good to go. Now we need to pull in that JSON data, so just like we were doing before. This time I'm going to say us.json, and I have a function here that's going to get the JSON data. And inside of here I basically need to add that path, so I'm going to say svg.selectAll("path"). Data is the JSON. Now, it's a JSON object, so inside of there remember the format we have we were looking for features, and those in this case are going to be states. Then we'll add the .enter function so it'll create them where they're missing, and we're going to append a "path". We'll set some attributes, primarily "d", path. So up above we have our path function, and here what we're doing is we've bound the data to our SVG, and we're telling it to create a path, and the data for that is going to be generated from this function call here. And we'll just give it a gray fill that we tend to like, so this is fill, and it's that #666666. Okay, so once we've got that and we've got this guy, let's see what we get. I'll Save this, Preview, and there you go. We have our map simple as that. Now, I can adjust this, and you'll see what'll happen is if I were to drop this down to 300 it'll chop it off a little bit. And you may be wondering did the people that build this are they somewhat self-centered being from California, and that's kind of where the map seems to be centered? Well, what actually is happening is it's drawing these shapes, but they're just outside of our SVG window. So just like when we scale our axes for our lines or any of the size of our bars, we need to essentially scale the size of our shapes here so that they fit within the space that we want them to, very important when it comes to making sure that your design works when you display it on the web. So let's do that. Let's now make this actually fit within something that we want to see. I'll save this one as map-1, and that'll be in the downloads, and then for the second one I'll just go ahead and save it as map-2. And let's give ourselves a little bit more room. We'll make the width 500. Now the way that maps get scaled is called a projection, so we're going to create a new function called projection, and we have inside of our geo library something called albersUsa, which is the default projection for D3. And we're going to translate this, and translate as you recall modifies the shape somewhat. And we're going to take the width of it divided by 2 and the height of it divided by 2. So, we're basically just going to chop it in half, and then we'll scale that to 500, which is the width of our object here. Now, once we've got that, when we generate the path we need to call our projection, so we say .projection(projection). And you can get better with naming it over time. For now it's pretty simple. So, before I save this guy, make sure that we have our translation as an array, so we've got to put the brackets around it. We'll hit Save and Preview, and there you go. We have something that looks like a map that we can at read. If you were to open this in a full window, what you'd get is the full size map like that. So, those are the basics of drawing a map using D3. DEMO: Adding Color Now let's take a look at making our map a little bit more interesting and trying to drive a little bit more meaning form it than just a shape by adding some color. This is what's known as a choropleth. First I'll take a look at an example here by Mike Bostock. This is one that's on the bl.ocks.org, that's bl dot ocks dot org, and you can take a look at lots of different examples here. This one is kind of fun, and it explains this idea of a choropleth and how you can use color to draw different shapes. Here I believe he's actually drawing different counties, and it creates this interesting effect through changing the color, so I think one thing that we need to be cautious of is actually using color a little bit too much. Like while this is interesting, I'm not sure if it's actually telling us much about the information. So, let's take the map we've been working on and see what we can do here. Now, first we have to think about how we're going to color something, and the way I'm going to do it is by using another file that has sales by state, so if you download the demos.zip you'll see something in there called sales by state. I'm going to jump over to the Could9 IDE now, and here's the map we've been working on. I'm going to go ahead and upload my sales by state, and once I've done that I can go ahead and crack this guy open and take a look. You can see what I have in here are state and sales separated by a comma, so we know we're working with the CSV. When we look at the data we see that there are float values here, so I have to be cautious about how we handle that. Let's go take a look. Everything seems to be formatted correctly. We have the full state name, which is good because it happens to match our JSON file that we generated earlier. I'm going to go ahead and save this as a different name now, and I'll clear my Preview window out of the way. I'll clear my workspace file so now we're working with a full screen view here. So what we want to do is we want to take our JSON which has the actual shape information that's stored in the features, and we want to color basically those paths that get drawn. Instead of just calling out a specific function, we actually want to use the data, so I need to actually first pull the data in and then call this other function, which pulls this data in, and then I'll show you a neat way to actually join those two data sets together. So, here I'm going to add d3.csv because we're working with a CSV file; state-sales.csv, that's our file that's living in there; we're going to have a function callback, which is going to receive some data; and we need to make this JSON call inside the CSV call. That way both of the data sets are being worked on at the same time, and they're not trying to execute and run past the other one. So like before when we had scales and domains we actually need something to tell our viz how to color it by. So, I'm going to go jump outside of my CSV call here and actually create a color domain. I'm going to use a scale function and make it a linear one like before. And this is where we get to have some fun, and we get to actually pick in our colors. If I look at the one that Mike Bostock did, it's this kind of turquoise to tan, and there are lots of different options. One of the best places to find this is on colorbrewer2.org. This is a website from Cynthia Brewer, and what they did is they made some really good ways for us to use different colors, and it has it all kind of baked out for you. There's a lot of science that goes into what colors are complimentary versus contrasting and which ones you should use where. So, if we wanted to try to build one like Mike's, we could use this color palette. I'm more of a fan of orange, so maybe I'll choose this one, and I want to see maybe five or six different classes so I have a bit more variability. And now I have all of my RGB codes here, which is what I need to put into my scale, but instead of having to type those out manually there's this neat little EXPORT function here. I'll click on that, and then you can see for JavaScript it has it right here. I'll just copy that, go back to my viz, I'm going to give it the range, and inside of there I'll just paste my RGB codes. So that was easy. I was able to go find a color palette, one that makes a lot of sense from a neuroscience perspective, and then actually just get the code directly out of it and past it in, so colorbrewer2.org is your friend. It's also in the list of references for this file here. So now I'm going to go back, and I want to actually use that color now to create the domain. Just like before when we did a min and a max value to scale our viz to fit within the window, here I'm actually going to say 0 to start, and I'm going to do what I did before d3.max(data, function(d) (return d.sales;). Cool. So what that's going to do, that's going to take the CSV file, which is being loaded into this data variable, and then it's going to run a function and parse through that and look at it and find the max value in that array that is in the sales column. Make sure I have my semicolon there. Okay, so now I've got my domain. Now let's jump into our JSON function. So, we've got these guys working together, and I'm just going to have a simple for loop. There are probably a dozen different ways to do this, but I'm just going to have one here for i=0, so that means my iterator is going to start at 0. It's going to be less than the length of my data set, and I'm just going to say i++, which means every time it runs through it's going to iterate on that and increment that by one. I'll create that as a function, get my curly braces. I'm going to keep the path call outside of that because this is really just to join the data together, and in here what I can do now is I can have the state coming from the JSON and the state coming from the CSV. The first loop is going to go through the actual CSV file, and then the next one is going to go through the JSON file. So on this outer one I'm going to say var salesState, give it a variable name, = data(i) for the row we're on, and state. So, that should give me the state for this row that we're parsing. And then we need to get the value out of it, so this is the salesValue, and we know it's a float, so remember we have to do parseFloat, data,(i) for the row we're on, and sales. So now that we've got the state and the sales amounts loaded, let's go run through the JSON file and try to find the corresponding value there. For var, I'll give this one a J for JSON, =0; j