This is the first of two tutorials on adding hover interactions to interactive maps using free, open-source tools.

Today I wanted to break down the step-by-step process of how I created this interactive, chloropleth map of Georgia population change from 2010 to 2011. While today’s tutorial doesn’t cover how to add hover states or moving tooltips to your maps, such as those you see in this iteration of the same map made using CartoDB+Leaflet, I’ll cover how to add those more advanced features in a later post (they require using a few more tools and doing a bit more programming). For now, though, we’re going to stick to the basics of simple hover interactions. Here’s what you’ll need to follow along:

• Mac OS X
• QGis (free)
• Tilemill (free)
• A free MapBox account where you will host your MBTiles
• Google spreadsheets (or Microsoft Excel, if you’re more comfortable with that)

The first thing you’ll need to do is locate your data and parse it down to a format in which it can be easily visualized. For this example, we’re using population estimates from the U.S. Census Bureau, which, luckily for us, has already done most of the heavy lifting analysis-wise. Download the data for Georgia or whatever other state you wish to visualize as a comma separated value (.csv) file from this page. Now you’ll want to turn that .csv file into a Google spreadsheet. You can do this from the Google Drive dashboard by selecting Create>Spreadsheet, then choosing File>Import once your new spreadsheet opens. Locate and upload the .csv file. Select “Replace current spreadsheet” and set “Comma” as the separator character. Voila. Your data should then appear for each county. Here’s what the correct options for that will look like:

Given that each county has a different base population, the only standar way to compare en masse how many residents each county gained or lost is by calculating the percent change (see here to figure out how to calculate that in Excel or Google Spreadsheets). In this example, however, the Census Bureau has already calculated out the percent change for us, making our job that much easier. Delete all the unnecessary columns from the spreadsheet, leaving only the county name in the first column, the 2010 and 2011 population totals in the second and third columns, and the “Percent” value in the fourth column (the “Percent” value refers to the percent change between the two years). Also make sure to delete any extra rows at the top or bottom of the spreadsheet, so that the first row contains the column titles, with the first county beginning on the second row and ending on the last row. Highlight the column containing the county names and select “Data>Sort sheet by column, A-Z.” This will put the entries in alphabetical order.

If you happen to get a weird period (‘.’) preceding each county name as I did, you can get rid of it pretty easily by performing the following steps:

1. Insert a new column to the right and, assuming your first county name begins in cell A2, enter the following formula into B2: =MID(A2,2,LEN(A2)). This function deletes the first character of the county name –– the unwanted period –– automatically.
2. Copy and paste the new B2 cell into the rest of the rows to apply the same formula throughout the entire spreadsheet.
3. Copy the new period-free column you just created and paste it as unformatted text into another new column to the right by selecting “Paste special>Paste values only.” This formats the data in new column as values only so that it they won’t include a formula that depends on the incorrectly formatted column to work.
4. Delete the first two columns so that the first column now becomes the county names only, free of the preceding periods.

We’re almost finished getting the data ready. All we have left to do is add in a column containing the official county codes for each county so that we’ll have a common attribute with which we can merge the spreadsheet with the geometric data later. Download this .csv from the Census Bureau, which contains the county codes for all 50 states. Open it as a Google spreadsheet and delete all the rows except for the ones for the state you’re visualizing. With only the rows for the state at hand remaining, highlight the “ctyname” column and select “Data>Sort values from A-Z.” This should reorder the spreadsheet to be exactly the same alphabetic order as the spreadsheet with the population totals. Copy the “county” column containing the numeric county codes, which should now also be arranged in numeric order, and paste it into a new column in the spreadsheet with the population totals, which should also be sorted alphabetically. This should now give you the correct county codes for each county in a new column. Title that column simply “COUNTY” (all-caps). For an idea of what things should look like at this point, check out my Google spreadsheet here, or see the following screenshot:

One last thing to keep in mind: If your county codes include values less than three digits, which they probably will, make sure any values less than three-digits long have 0s preceding them to force them to be three digits long (i.e. ’001′). That way, it will match up with the three digit values in the geometric data later on in the process.

Now that we have our population data ready, let’s download the corresponding geometric county polygons from the Census Bureau here. For this tutorial we’ll be using the shapefiles (.shp) format, so make sure to select the shp.zip option on the download page. I obviously used Georgia in this example; feel free to choose whatever other state you desire, so long as you follow the exact same instructions. Unzip the archive to your computer. You should see three files in the new unzipped folder: a .dbf, a .shp and a .shx. All we’ll be need for this tutorial is the .shp file.

At this point, you should have two different files: a Google spreadsheet of data formatted something like this, and a shapefile of corresponding county polygons. The next step will be to take the population data and bind it to the shapefile so that the two match up.

Fire up QGis. Select the “Add Vector Layer” option from the top of the window and locate the .shp file you downloaded earlier. Open it in QGis and you should see a nice outline of your state that looks something like this:

Now go back to your newly created Google spreadsheet and export the data as a .csv by selecting “File>Download data as.” After downloading the .csv, rename it to something simple like “georgia.csv.” The next thing we need to do is import the .csv into QGis. But before we can do that, we need to create a new ‘.csvt’ file by the same name and in the same directory as the .csv that will tell QGis what type of data each column is (string, number, real, etc.). For this example, your ‘.csvt’ file will look something like this, with a data type defining each column of the .csv:

If you’re having trouble with this part, download my .csvt here. Should you need to adjust the data types, just force open it in TextEdit and change them. The main two things to make sure is that the COUNTY column is defined as a string and the PERCENT column is defined as a value.

Once your .csvt file has been placed in the same directory as your .csv, go back to QGis and add a new vector layer just as you did before with the shapefile. This time, locate the .csv file and open it. QGis should then automatically detect the .csvt file in the background and assign the appropriate data types to each column in the .csv. To make sure this worked correctly, you can control-click the new .csv vector layer and select “Properties>Fields” to check that each field has the appropriate data type.

Now you can get down to the business of binding the data from the .csv to the shapefile. Select the shapefile layer and go to “Properties>Joins.” Add a new vector join, setting the .csv as the join layer and both the join field and the target field as COUNTY, like this:

Applying the join should merge your spreadsheet and shapefile, binding the population data to the polygons using the shared “COUNTY”attribute that contains the matching county codes. To check and make sure everything worked correctly, control-click the shapefile vector layer in QGis and select “Open attribute table.” You should see the population data attached as columns to the end of the attribute table.

Once you’ve ensured that the data has been attached to the polygon vector layer, you can now export the new shapefile by control-clicking the shapefile vector layer, selecting “Save as,” and saving the layer in the ESRI Shapefile format. Now you’re ready to compress the shapefile package into a .zip and import it into Tilemill. where you can style it, add interactivity and more.

#georgia {
  line-color:#565A5D;
  line-width:0.5;
  polygon-opacity:1;
  polygon-fill:#ae8;
}

#georgia[PERCENT<=5.5] {
   polygon-fill:#558740
}
#georgia[PERCENT<=4] {
   polygon-fill:#609948
}
#georgia[PERCENT<=3] {
   polygon-fill:#6BAA50
}
#georgia[PERCENT<=2] {
   polygon-fill:#7DB863
}
#georgia[PERCENT<=1.5] {
   polygon-fill:#9FBA80
}
#georgia[PERCENT<=0.75] {
   polygon-fill:#B8CCA1
}
#georgia[PERCENT<=0.3] {
   polygon-fill:#D9DCC4
}
#georgia[PERCENT<=0] {
   polygon-fill:#F4E5E0
}
#georgia[PERCENT<=-0.5] {
   polygon-fill:#E6C5BB
}
#georgia[PERCENT<=-1] {
   polygon-fill:#D8A696
}
#georgia[PERCENT<=-1.75] {
   polygon-fill:#CC8472
}
#georgia[PERCENT<=-2.5] {
   polygon-fill:#BC5942
}
#georgia[PERCENT<=-3.9] {
   polygon-fill:#B34027
}
#georgia[PERCENT<=-5] {
   polygon-fill:#A1280F
}

As you can see, the first set of paramaters defines the style attributes of the entire polygon, including the line width and the opacity of the #georgia layer. The following sets of paramaters all set the fill color of individual ‘buckets,’ sorted by the value of the PERCENT field.

Now, add layers for the surrounding geographical and cultural features, including roads, populated places, rivers, and surrounding states. When selecting the file source for your new layer, click “Browse>MapBox” to add predefined features from MapBox. Use the same method to style those layers as you did with the primary shapefile layer, pressing Command + S to save the map and refresh the view. Here’s what my styled map ended up looking like:

4. Adding the mouseover tooltips
In Tilemill, navigate to the Templates panel. Select “Teaser.” This is the field that defines what appears in the tooltip that appears when the user moves the mouse over each polygon. Basically, all you have to do is format the field to correspond to the values in your data.

Here’s the formatting I used:

{{{CountyName}}}
Percent Population Change, 2010-2011:
{{{PERCENT}}}%
2010 Population:
{{{2010}}}
2011 Population:
{{{2011}}}

That produced the following tooltip:

As you can see, I used simple HTML markup, including the bold and horizontal line tags to add style elements. Each instance of {{{TEXTHERE}}} corresponds to a column name from the shapefile, and calls it up in the tooltip. So, to call up the percent change, I typed {{{PERCENT}}}, which refers to the PERCENT column from our data. Upon saving this, you’ll see that the tooltip appears however you format it. The possibilities here really abound.

You can add a custom legend to map in much the same way. While you’re still in the Templates panel, select “Legend.” This allows you to use basic HTML/CSS both to define the structure and style of your legend, as well as the position of both the legend and the tooltip. Here’s a look at how I formatted the “Legend” field to produce my map:

Estimated Population Change, 2010-2011

  

    
<-5%
    
-5--4%
    
-4--3%
    
-3--2%
    
-2--1%
   
-1--0.5%
   
-0.5-0%
   
0-0.3%
   
0.3-0.8%
   
0.8-1.5%
   
1.5-2%
   
2-3%
   
3-4.5%
   
>4.5%

  

Source: U.S. Census Bureau, 5/17/2012.

If you know rudimentary HTML/CSS, it’s pretty easy to tell what’s going on above. The background attribute corresponds to the color ramp I defined earlier in the style.mss field of Tilemill, and the HTML sets up the structure of the legend. After the HTML, I set some basic CSS styles, including the positioning of the legend and tooltips using the “margin” attribute and the font-size of the legend text. Save all that, and here’s a glimpse of what you get:

After you have the map styled to your liking in Tilemill, it’s a fairly straightforward process to get the map on the Web. First, in the top right of Tilemill, select “Export” and choose “MBTiles.” This is the format for the tiles that you’ll need to host the map on MapBox. Once the export panel appears, set the starting center point (for my map, it was -83.562,32.9349,6), as well as the bounds. Keep in mind that the smaller the bounds, the less tiles your map needs to render and, thus, the faster it will load. Only include the bounds you need in the map so as to minimize load time. Give the map a name, a brief description, and be sure to attribute the data source. Once you’re finished, click “Export” and save your map in the directory of your choice. The resulting .MBTiles file will now go directly into MapBox, where your map will be hosted.

The final step in making your map viewable by the world involves uploading the .MBTiles file you just exported to MapBox. Create a MapBox account and select the option to upload a new map. Locate your .MBTiles file and upload it to MapBox. Once it’s finshed you’ll see a preview of the map that looks like this:

Enable the “Map Details” view. Set the map to “Public,” and make sure the Zoom, Legend, Scroll zoom and Tooltip options are selected. Deselect the “Share links” option. Then click embed, and copy the embed code to use the map wherever you please.

Just to give you a live view of how the finished map looks and behaves:

And there you go. Using Tilemill to create hover-enabled maps is easy and flexible, but it has its limitations. Because the program renders data using the UTF grid-system, it doesn’t send vector data to your browser, meaning you can’t use JavaScript or CSS3 to spice things up. The next step? Adding hover states and moving tooltips to create maps like this for polygons and this for markers. Until then, feel free to ask any questions or point out any problems you may run into. Happy mapping!

Leave a Comment

Although geared primarily toward the production of static graphics for print publications, Dona M. Wong’s The Wall Street Journal Guide to Information Graphics (2010) provides a wealth of salient and time-honored tips and guidelines that any student of data visualization would be well-advised to follow. At the heart of Wong’s book is the notion that data integrity trumps all else, and no matter how aesthetically pleasing or visually powerful an information graphic may be, if it doesn’t communicate clear and accurate data to the reader/user, it doesn’t do its job.

In the first two chapters of The WSJ Guide, Wong, a former student of data viz extraordinaire Edward Tufte, addresses the topic of charting. From a theoretical standpoint, Wong lays out four principle steps to the charting process:

1. Research: Find your data source, and ensure that it’s timely, authoritative and free of bias.
2. Edit:  Figure out what the data says (essentially, determine what your story is), and conceive of how best to boil that data down in a way that’s simple enough for your intended audience to understand without skewing its meaning.
3. Plot: Determine the appropriate chart type for your data (e.g. bar, column, line, pie, stacked bar, etc.), choose the right settings (scale, increments, axes, etc.), labeling the chart (e.g. legends and source lines) and pick the best color and typography combinations to accentuate your key message.
4. Review: When you’re done, ask yourself the following questions: Does the data match up with what external sources say? Are there any outliers? Does the chart make sense? What would the average user/reader think upon first seeing the chart?

Regarding the finer points of charting, Wong does an excellent job at pointing out the various dos and don’ts of the presentation process.  She sets forth clear guidelines about when to use what type of chart. For example, when dealing with change over time, Wong says to always use a line chart instead of a bar chart, as bar charts should ideally be reserved for comparing several different series of data. Also, Wong asserts, pie charts usually aren’t as good of a choice for displaying complex data as bar or line charts, primarily because they make it harder to discern discrete differences in size (later, she flat-out dismisses the donut pie chart for the same reason). A few of her other tips I found particularly relevant included: (a) avoiding high-contrast color schemes that draw attention away from the data, (b) shying away from icons with high detail so as to avoid visual overload, (c) never, under any circumstances, add cloying shadow or 3D-effects and (d) never rely on zebra patterns, dotted lines or other fancy methods of labeling. “A chart is not a piece of fine art,” Wong says.

Most importantly, Wong sets forth some other general principles to help designers avoid creating misleading charts. For example, when creating a bubble chart, always plot the bubbles by area, not radi. Also, never plot two different data series on noncomparable scales, and when creating bar charts, always start at the zero baseline. Other steps to ensuring data integrity include putting numbers into their appropriate context (comparing apples to apples), holding off on rounding until the end of the data analysis process and avoiding charting predictive numbers alongside actual ones. As Wong so eloquently puts it, “Unlike a misspelled word in a story, one wrong number discredits a whole chart.”

Although the new addition of interactivity to chart design adds another layer of complexity to the visualization process that Wong doesn’t address here, most of the guidelines she sets forth hold true in both static and dynamic mediums. Yet it would be interesting to hear what she has to say about the vexing question of when and when not to add static labels to interactive charts…

Leave a Comment

In keeping with my recent spate of mapping nerdiness, I decided to take an interactive map I produced last month displaying statewide annual population changes a step further by adding mouseover/hover capabilities. Here’s the hover-y, nicely-colored chloropleth map I came up with. But before I get into the nitty-gritty of how I created the map –– which I’ll explain step-by-step in a later post –– let me exercise a bit of self-indulgence by defending my growing belief in the need for hover capabilities when visualizing geographic data.

Not too long ago, I was an avid believer in the no-frills, less-interactivity-is-more approach to mapping geographic data, espoused by the brilliant Brian Boyer (@brianboyer), News Applications Editor for NPR and a former member of the News Apps Team at The Chicago Tribune. Boyer’s argument for the need to keep maps simple –– like they used to be back in the days of ink and paper –– certainly has its merits. After all, the process of bringing a physical map closer to one’s eyes to get a better view is a natural, timeless user interaction, and maps like this one, which Boyer produced during his time at The Tribune, are far more intuitive in communicating information upon first glance than many of the infoWindow-laden Google maps being produced by news organizations these days, many of them simply for the sake of being called ‘interactive’ (for those of you fortunate enough not to be mapping nerds, infoWindow is just Google-speak for the clickable popup boxes you see in Google maps).

But Boyer’s minimalist mapping aesthetic only really works when you have one or two pieces of textual data you want to display for each geographic area. What if you have multiple pieces of information you want to display for each polygon, such as in this snazzy map from The Texas Tribune? Or, less likely but equally problematic, what if you need to bind non-textual data to your geographic polygons, such as images or Google Charts? In cases such as these, you’re going to need to provide some sort of interaction that allows the user to expand and collapse the data for each area individually, or you’ll just end up with a chronic case of visual overload.

Not to mention, on a more abstract level, studies have repeatedly shown that users tend to spend more time on applications that provide direct feedback based upon their actions, even if that feedback sometimes makes their ability to consume information at first glance less efficient (see Donald Norman’s 2005 book Emotional Design: Why We Love (Or Hate) Everyday Things, in which Norman asserts that the feeling of emotional satisfaction and empowerment users receive from triggering an action not only puts them in a clearer state of mind, but also makes them more engaged in the information at hand). So, if we’re trying to communicate geographic data to users as effectively as possible, it only makes sense that we’d want to have a certain degree of user interaction –– both for the sake of preventing visual overload and for making users feel more engaged. Such is the logic behind clickable infoWindows.

Still, clickable popups leave us with another problem: Users have to make the conscious and deliberate effort to click a polygon to see the data for that geographic area. Requiring clicks may sound like a trivial task to the designer or journalist-programmer, but for the short-attention span user, it can be an awful lot to ask for. To be fair, however, click-triggered popups may not be much of a problem for maps with only a few dozen polygons. But for maps with hundreds of small polygons –– say, census tracts or zip codes –– it can be very tedious to click the right polygon without first having to zoom into the map so far that you lose sight of the broader context.

That leads me back to a conversation I had a couple of months ago with a friend of mine from Columbia’s J-School, Michael Keller (@mhkeller), who’s now working as the Senior Data Reporter at The Daily Beast. Michael insisted to me after a Hacks/Hacker event that providing hover interaction for maps is almost always a good thing, because hovers require less work on the part of the user. I’ll admit that I was dubious at the time, thinking of hovers as often unwanted, accidental triggers that can be distracting to the map and data at hand. But lately I’ve come around to his way of seeing things. If implemented correctly (i.e. no flashy interactions that cover up other parts of the map), hovers are almost always a good idea. For example, this recent map of New York Stop-and-Frisk data that Keller produced for The New York World using CartoDB and Leaflet is so detailed that it couldn’t possibly have worked without infoWindows, and would have been unwieldy if it relied on click-triggered interaction. By including floating mouseover capabilities, the map allows the user to scan quickly through the chloropleth map to see individual Stop-and-Frisk data from each block, without having to attempt to click through minute geographic areas.*

I’m certainly not advocating interaction for interaction’s sake (although such a case could be made, given the dynamics of the Web). But I am saying that hovers give more immediate visual feedback than click-triggered events, especially in maps. Hovers help draw users into the data without requiring them to seek it out consciously–– almost like a catchy lede would in a print narrative. So for the time-being, I’m pro-hover.

*Keller later messaged me letting me know that some examples which better illustrate the power of hovers include this map and this map, both of which use hover functionality to help highlight the effects of proposed redistricting efforts in the New York State Senate. What you’ll also notice about Keller’s maps is that they include hover states, which I also think is a necessity, especially for maps that include lots of small polygons.

Leave a Comment

An ever-so-sleek responsive portfolio site I designed for a friend in less than 20 minutes using Skeleton as a foundation.

With all this talk lately of the new era of responsive design, I realized today that I’ve yet to create anything that’s actually responsive. Given that I’ve only pondered using it in the implementation of complex, database-driven news sites, the task of tweaking every level of CSS to fit perfectly into a responsive grid system has so far seemed too daunting to tackle.

But I got curious this afternoon, and found my way upon a cool new library called Skeleton, which bills itself as a “a beautiful boilerplate for responsive, mobile-friendly development.” Essentially, Skeleton is a collection of CSS and JS files that makes the mystery of responsive design seem a little less illusive. Upon uploading the package to my server, I was pleased to find a neatly-coded, easy-to-understand responsive site that I could play around with and tweak to my own liking. I ended up adjusting the grid size and performing some minor customizations to the underlying Skeleton structural CSS, but other than that, the development kit was, as it promised, “style agnostic.”

Since I didn’t have any specific projects I needed to be working on today, I decided to give Skeleton a whirl by designing a new online portfolio as a favor for my friend Daniel Medina. I’m not kidding – within 10 minutes I’d coded a fully satisfactory responsive portfolio site that looks beautiful on my iPhone and tablet.

So, if you’re feeling experimental, try Skeleton out. It’s exciting to see how quickly this technology is catching steam, and to see the actual workings of the responsive CSS in action firsthand. My next goal is to integrate a responsive layout into a more-powerful database-driven site, perhaps one designed in Drupal.

Leave a Comment

"Raw data is both an oxymoron and a bad idea; to the contrary, data should be cooked with care." ~ Geoffrey Bowker, 2005

Bowker is spot on. I can’t tell you how many times I’ve had to make order out of chaos from “raw data,” i.e. unintelligible, inaccurate spreadsheets.

Leave a Comment

I’ve been interested lately in finding examples of online-only, collaborative, non-profit newsrooms who’ve utilized the power of data visualization techniques to give added value to stories that otherwise wouldn’t necessarily be unique, and in doing so beat out legacy news organizations who published a text narrative alone. Take, for example, this data-rich story and interactive map displaying statewide testing results published by NJSpotlight Friday. While the news that only 8 out of 10 graduating seniors had passed New Jersey’s current standardized test in 2011 was widely reported across the state last week, including by the Star-Ledger in Newark and by The Press of Atlantic City, only NJSpotlight took advantage of the story’s strong data element to produce a more concise, data-driven visual narrative.

So NJSpotlight obviously deserves kudos for the gap they’re filling in New Jersey journalism. What’s more, the job they did on the interactive map was fairly sophisticated (I’ve still yet to figure out how to overlay such a highly customized legend onto a Google map). But as is always the case with any deadline project, there’s room for improvement. Let’s take a look at the good and the bad of NJSpotlight’s Friday package on state test results.

First off, this is a classic example of a story where county and/or municipality polygons with a colored fill layered on top of a Google Map brings new insight to a widely reported story. Not only can we immediately see from the map that most of the state fell within the 75 to 90 percent range in terms of passing rates, but we can also clearly tell that the north-central region of the state, particularly near the Pennsylyvania border, earned significantly higher scores than the rest of New Jersey. The pop-up table with information on vocational and charter schools also adds an additional layer of nuance to the piece, and does a decent job of displaying the numbers in a table-like format.

The map colors follow a somewhat logical pattern, with green representing high-passing rates and red representing lower passing rates. But the orange, yellow and blue colors that fall in the middle do more to obfuscate than they do to help visualize. Without referencing the legend, how is the user supposed to know that blue is better than yellow, or that green is better than blue? What’s more, the combination of such bright shades of colors from opposite ends of the color spectrum makes the map less pleasing on the eye than it would’ve been had the designer chosen more subtle, complimentary shades. I understand the desire to have red represent ‘negative’ and green represent ‘positive,’ but if NJSpotlight had went with a graduated color scale from red to green, with neutral-based mid-values such as the ones I used in this recent map on nationwide obesity rates, the map would’ve been not only aesthetically more appealing, but easier to read.

Another minor criticism I could make of the map is the designer’s decision to set the polygon-fills to solid colors with 100 percent opacity. This obscures the underlying view of which cities and townships each shape contains. While I understand the thought-process that likely went into this decision – it’s hard to make disparate colors, especially orange and yellow, stand out when layered on a slightly orange-tinted map with road and highway features – the designer could have easily used a tool such as colorbrewer2.org to generate some nice equidistant colors that would’ve looked fine set to an opacity of about 50 or 60 percent.

For the sake of thoroughness, I also want to address the included table that contains the charter and vocational school data. I like the fact that NJSpotlight chose to alternate the shades of background color for every other row in the table. It helps distinguish each row more easily from its neighbor, and gives an extra visual attribute to what easily could have been a stale grid. My only recommendations might be to split up every five or six rows with a dividing line, and to incorporate a bar chart, if possible, for one of the values in the cell (presumably, the most important value, the percent passing column).

Leave a Comment

I’ve been tinkering around with some new mapping tools lately, and figured I’d put them to good use by displaying the 2011-2012 population estimates released last week by the U.S. Census Bureau. The inherently geographical nature of the census makes it a data set just begging to be mapped.

Rather than the de facto Google Maps JavaScript API V3, I decided to go with CartoDB and Leaflet to see what I could produce.

As I mentioned in a recent post, CartoDB offers an excellent Fusions-esque interface, although it allows for far less front-end customization and requires more beneath the hood programming. Nonetheless, CartoDB can make pretty maps right out of the box, which you can then fully customize using the CartoDB API and basic SQL statements. There’s one caveat, however: The service only allows you to upload 5 tables for free. That could be a dealbreaker for cash-strapped news organizations and freelance data journalists.

Anyhow, I downloaded a .zip shapefile package of all 159 Georgia counties from the U.S. Census Bureau, then brought the package into CartoDB using the service’s default upload interface. Using Excel, I calculated the percent change from the most recent population estimates to last year’s estimates. I then added the resulting values as a column in my CartoDB table, which you can see here.

After playing a bit with the API, I was able to format a diverging chloropleth map from my table with the following style parameters, written using 0to255 to ensure an equidistant color scheme:

#statewidepop {
   line-color:#FFFFFF;
   line-width:1;
   line-opacity:0.52;
   polygon-opacity:1;
}
#statewidepop [percent_change<=5.5] {
   polygon-fill:#558740
}
#statewidepop [percent_change<=4] {
   polygon-fill:#609948
}
#statewidepop [percent_change<=3] {
   polygon-fill:#6AA84F
}
#statewidepop [percent_change<=2.25] {
   polygon-fill:#BECFA8
}
#statewidepop [percent_change<=1.5] {
   polygon-fill:#D0D8BB
}
#statewidepop [percent_change<=0.75] {
   polygon-fill:#B8CCA1
}
#statewidepop [percent_change<=0.3] {
   polygon-fill:#D9DCC4
}
#statewidepop [percent_change<=0] {
   polygon-fill:#D3BAAF
}
#statewidepop [percent_change<=-0.5] {
   polygon-fill:#E7D1C5
}
#statewidepop [percent_change<=-1] {
   polygon-fill:#D8A696
}
#statewidepop [percent_change<=-2] {
   polygon-fill:#C36E59
}
#statewidepop [percent_change<=-3] {
   polygon-fill:#BC5942
}
#statewidepop [percent_change<=-4] {
   polygon-fill:#B34027
}
#tl_2009_13_county[percent_change<=-5] {
   polygon-fill:#AB2B10
}

Check out the resulting map:

The map above shows the percent change in population from July 2010 to July 2011 in all 159 Georgia counties, as estimated by the U.S. Census Bureau. The darker the green, the higher the positive percent change. The darker the red, the higher the negative percent change. Click on a county to see its percent change.

Pretty nice, huh? But what if I want to customize the style of the pop-up windows or perform more advanced functions like creating custom image markers or switching between layers? That’s where Leaflet, an open-source JavaScript library, comes in handy.

Using the Leaflet library

The map above displays the estimated percent change in population of various midstate counties between July 2011 and July 2012. The greener the county, the higher its percent increase. The deeper red the county, the higher its percent decrease. Click on a county to see more precise totals, or select a group from the dropdown in the top right corner for a breakdown of the population changes by race.

To get a wider range of flexibility, I called up a segment of the statewide data – the counties within the Macon/Warner Robins metropolitan area – using the Leaflet javascript. Leaflet allows you to reference layers from CartoDB or Google Maps from within its API, making integration a breeze. All you have to do is reference a few lines of code and your CartoDB data will appear as a layer on your Leaflet map automagically. But even on its own, Leaflet is pretty robust, especially for being so lightweight.

In the map above, I took the county shapefile package from earlier, converted it to GeoJSON using QGis, then, following these parameters, called up the GeoJSON data for the selected counties using the Leaflet script. For the underlying map tiles, I created a custom style using Cloudmade, then referenced it using my API key and the following line of script:

var cloudmade = new L.TileLayer('http://{s}.tile.cloudmade.com/41fc7e18cef34d6fb34756efd8240787/63501/256/{z}/{x}/{y}.png',

Because I also wanted to show a breakdown of the data by race for the same geography, I added in a custom control menu that allows the user to switch between layers for easy comparison. In addition, I styled the popup to my liking, with green and red values to connote increase and decrease.

From there, I was able to add in the additional data sets of population change by race. For each demographic group, I created a corresponding layer group. Each layer group contained the data as well as the appropriate styles and colors. See the source code below:

var totalLayer = new L.LayerGroup();

citiesLayer.addLayer(geojsononeLayer)
           .addLayer(geojsontwoLayer)
           .addLayer(geojsonthreeLayer)
           .addLayer(geojsonfourLayer)
           .addLayer(geojsonfiveLayer)
           .addLayer(geojsonsevenLayer)
           .addLayer(geojsoneightLayer);

var whiteLayer = new L.LayerGroup();

whiteLayer.addLayer(geojsononewhiteLayer)
           .addLayer(geojsontwowhiteLayer)
           .addLayer(geojsonthreewhiteLayer)
           .addLayer(geojsonfourwhiteLayer)
           .addLayer(geojsonfivewhiteLayer)
           .addLayer(geojsonsevenwhiteLayer)
           .addLayer(geojsoneightwhiteLayer);
var blackLayer = new L.LayerGroup();

blackLayer.addLayer(geojsononeblackLayer)
           .addLayer(geojsontwoblackLayer)
           .addLayer(geojsonthreeblackLayer)
           .addLayer(geojsonfourblackLayer)
           .addLayer(geojsonfiveblackLayer)
           .addLayer(geojsonsevenblackLayer)
           .addLayer(geojsoneightblackLayer);

var asianLayer = new L.LayerGroup();

asianLayer.addLayer(geojsononeasianLayer)
           .addLayer(geojsontwoasianLayer)
           .addLayer(geojsonthreeasianLayer)
           .addLayer(geojsonfourasianLayer)
           .addLayer(geojsonfiveasianLayer)
           .addLayer(geojsonsevenasianLayer)
           .addLayer(geojsoneightasianLayer);

var otherLayer = new L.LayerGroup();

otherLayer.addLayer(geojsononeotherLayer)
           .addLayer(geojsontwootherLayer)
           .addLayer(geojsonthreeotherLayer)
           .addLayer(geojsonfourotherLayer)
           .addLayer(geojsonfiveotherLayer)
           .addLayer(geojsonsevenotherLayer)
           .addLayer(geojsoneightotherLayer);

map.addLayer(citiesLayer);

var overlayMaps = {
    "Total Population": citiesLayer,
    "White": whiteLayer,
    "Black": blackLayer,
    "Asian": asianLayer,
    "Other/Mixed": otherLayer
};

var layersControl = new L.Control.Layers(overlayMaps);

map.addControl(layersControl);

All of this was accomplished using the built-in components of the Leaflet JavaScript API and only an hour or so of data analysis in Excel. Next goal? Adding mouseover capabilities.

Leave a Comment

Others may hate, but I’m a big fan of using bubbles to display data. When implemented correctly (i.e. scaled in terms of area instead of diameter), bubbles can be an aesthetically appealing and concise way to represent the value of data points in an inherently visual format. Bubbles are even more useful when they include interactivity, with events like mouseover and zoom allowing users to drill down and compare similar-sized bubbles more easily than they can in static graphics. So, when I was recently working on a class project on autism diagnoses in New York City, I decided to use bubbles to represent the percentage of students with individualized education plans at all 1250 or so K-8 New York City schools.

Almost by default, I turned to Google Maps JavaScript API V3, mainly because I’m quasi-familiar with its basic functions and event handlers (as I point out later in  this post, I didn’t realize that a nifty new service called CartoDB would have automated most of what I was trying to do, albeit without nearly the level of customization). Nonetheless, based on a tutorial from Karl Agius, as well as some infoWindow help from my data viz professor, Susan McGregor, I created the following interactive bubble map of NYC schools based upon the number of special needs, or IEP, students at each school. The larger the bubble, the greater percentage of special needs students a school has. Click here to see the map full-screen, or here to download a .zip of my source files for your own customization purposes.

Each bubble on this map represents one of New York City’s approximately 1,250 K-8 public schools, including charters. The larger the bubble, the higher the percentage of students with individualized education plans (IEP). Click on a bubble to find out more about the school, or click anywhere within a district boundary to see an overall average IEP rate for the district. Zoom and pan to see other parts of the city.

You’ll notice the opacity for the bubbles is set to 40 percent. This allows us to get a quick visual of the locations with the highest density of special needs students, given that those areas on the map will naturally be darker because they have multiple semi-opaque layers that overlap. Setting a low opacity also prevents overlapping bubbles from covering up one another. You’ll also notice that the map includes polygons for each school district, which you can click on to get an average IEP rate for the entire district. I decided against setting gradated fill colors for the school district shapes so as to avoid implying causation, as well as to lessen the visual clutter.*

Preparing the data

To create the map, I first had to download the underlying data from the New York City Department of Education database as a .csv, then import it into Excel to clean it up and leave only the relevant information. Although the dataset I had only included street addresses split into multiple columns, I was able to use the concatenate function in Excel to merge the street, city, state and zip columns to get a full street address. From there, I used my favorite batch geocoding service to convert the addresses into geographic coordinates that the Google Maps API can read. Check out my resulting .csv file here for an example. Then I imported the .csv into a Google Spreadsheet, and pasted the resulting spreadsheet’s URL into dataSourceURL field in the JavaScript of my main index.html file. Here’s how that looked in my code:

var dataSourceUrl = "https://docs.google.com/spreadsheet/ccc?key=0Au4D8Alccn4xdHNJdXJmeTdYcEtpRXE1QXRucWtEN3c";
var onlyInfoWindow;

google.load("visualization", "1");
google.load("maps", "3", {other_params:"sensor=false"});

google.setOnLoadCallback(getData);

Drawing the bubbles

Using the ‘Circle’ marker style from the Google Maps JavaScript V3 API, I set each row in the spreadsheet to draw a bubble whose size corresponded to its value. It works by calling up the ‘size’ column in my spreadsheet, which contains the IEP data I want to display, and then drawing an overlay of bubbles based upon those values. Credit again to Karl Agius for helping me figure out this part of the script.

// This first array is where you can change the size and styling of the bubbles.
function Bubble(data, options) {
    this.data = data;
    if (options) 
		this.options = options 
	else 
		this.options = {
			radiusForPercentage:1000,
			text:{
				visible:false,
				minimumZoom:13,
				maximumZoom:15
			},
			bubble:{
				fill:{color:"#709ED9", opacity:0.6},
				stroke:{color:"#709ED9", weight:1, opacity:0.3}
			}
		};
	this.totalSize = this.getTotalSize(this.data);
}

Bubble.prototype = new google.maps.OverlayView;

Bubble.prototype.onAdd = function() {
	for (var row = 0; row < this.data.getNumberOfRows(); row++) 
		this.drawBubble(this.data, this.options, row);
}

Bubble.prototype.draw = function() {
	if (this.options.text.visible)
		for (var row = 0; row < this.data.getNumberOfRows(); row++) 
			this.drawText(this.data, this.options, row);
}

Bubble.prototype.getTotalSize = function(data) {
    var totalSize = 0;
    
	for (var row = 0; row < data.getNumberOfRows(); row++)
        totalSize += data.getValue(row, 1);
                
    return totalSize;
}

Bubble.prototype.drawBubble = function(data, options, row) {
	
    var sizeOfLocation = data.getValue(row, 1);
    var percentageOfTotal = (sizeOfLocation / this.totalSize) * 100;
			
    var radiusForLocation = options.radiusForPercentage * percentageOfTotal;
			
    var marker = new google.maps.Circle({
        center: new google.maps.LatLng(data.getValue(row, 2), data.getValue(row, 3)),
        fillColor:options.bubble.fill.color,
        fillOpacity:options.bubble.fill.opacity,
        strokeColour:options.bubble.stroke.color,
        strokeWeight:options.bubble.stroke.weight,
        strokeOpacity:options.bubble.stroke.opacity,
        radius:radiusForLocation
 });
	// This last command actually places the Circle markers we just made on the map.
	marker.setMap(this.getMap());

You can change the default size of the bubbles with the radiusForPercentage value, and set the colors using the fill and stroke fields. You can also set a minimum and maximum zoom so that users can only see the part of the map you want them to see.

Adding interactivity

Next, to make the bubbles clickable, I added an on-click event listener that set each bubble marker to fetch the corresponding data from my Google spreadsheet and add it to an infoWindow. Here's what that looked like:

var contentString = '
School Name' + data.getValue(row, 0) + 'Percent IEP students:'+sizeOfLocation+'Percent free-lunch:' + data.getValue(row, 4) + '
';

var infowindow = new google.maps.InfoWindow({
content: contentString,
position: new google.maps.LatLng(data.getValue(row, 2), data.getValue(row, 3))

});

google.maps.event.addListener(marker, 'click', function() {
if(onlyInfoWindow != null){
onlyInfoWindow.close();
}

infowindow.open(this.getMap());
onlyInfoWindow = infowindow;

});

As you can see, I referenced each value in my spreadsheet by adding in its row number. Later, I went back in and added another row of data that displayed free-lunch statistics as well, which is why you see a reference to row 4.

Overlaying the school district polygons

To place the school polygons onto the map, I added a new overlay that pointed to a Google Fusions table containing the KML geographic boundaries. This is what that looked like:

var schoolDistricts = new google.maps.LatLng(41.850033, -87.6500523);

var layer = new google.maps.FusionTablesLayer({
  query: {
    select: 'geometry',
    from: '3621394'
  },
  styles: [{
  polygonOptions: {
    fillColor: "#FAFBFF",
    fillOpacity: 0
    }
  }
  ]}
  );

layer.setMap(map);
  }

The 'from' field should correspond to your Fusions Table's six-digit ID, and the 'select' field should reference the column in your table that contains the KML geographic data.

A simpler new tool to do roughly the same thing

Ultimately, however, it may have been easier to use the WYSIWYG editor from CartoDB, a premium mapping service with a front-end interface similar to Google Fusions Tables except that it allows you to draw bubbles without having to dig into the API. See what happened when I uploaded my spreadsheet to CartoDB and set similar style attributes:

Which map is better? Personally I think the hand-coded Google Maps API version is better in terms of aesthetics and usability. But was it worth the extra effort? You tell me.

*Jennifer LaFleur, ProPublica's director of computer-assisted reporting, later pointed out to me that I could easily add a gradated fill for each of the school district polygons based upon poverty levels, provided that I pick the right color-scheme to complement the blue bubbles. I guess I was just a bit leery from a design standpoint about what one color on top of another would look like.

Leave a Comment

A quick refresher from my data visualization professor here at Columbia a couple of weeks ago reminded me why I was forced to spend all those grueling hours calculating standard deviation back in high school.

See, when you’re using a data set to tell a story, the first step is to understand what that data says. And to do that, you’ve got to have a good idea of the range and variation of the values at hand. Not only can figuring that information out help you determine whether there’s any statistical significance to your data set, but it can also pinpoint outliers and possible errors that may exist within the data before you begin the work of visualizing it.

Thanks to powerful processing programs like Excel, we can figure out the variability of data sets pretty easily using the program’s built-in standard deviation function (remember this intimidating-looking equation from calculus class?). Still, it always helps to know how to calculate the information out by hand, if only to get a conceptual idea of why numbers such as the standard deviation (the variability of a data-set) and the z-value (the number of standard deviations a given value is away from the mean) even matter in the first place when it comes to data visualization.

So, to brush up on my formulas and also better understand the numbers behind an actual story assignment for one of my classes, I recently hand-calculated the standard deviation and z-values for a set of data on state-by-state obesity rates. From my calculations, I was able to use the standard deviation (3.24) to determine that, on average, most states fell within the middle of the bell-curve for the average national obesity rate (27.1 percent) . In addition, the z-values helped me understand which states stood out from the pack as possible outliers (Mississippi is by far the most obese with a 2.13 z-value, Colorado the least obese with a -1.9 z-value). To get an idea of how those formulas look hand-calculated in Excel, check out my spreadsheet here. And keep these formulas in mind while working on your next data story. They can potentially save you time and effort by helping you figure out what your data set says before you have to go through the often-lengthy process of visualizing it.

Leave a Comment

With the Pulitzer Price announcements coming up later this afternoon, you’d think I’d be writing about whose up for the “Best Deadline Reporting” or “Best Public Service Journalism” prizes. But instead I want to talk about a different media award doled out during the past week: BostonGlobe.com’s designation as the “world’s best designed website” by the Society for News Design. Put simply, I can’t say I disagree.

Yet before I divulge in my effusive praise of the folks in Boston, let me point out that I’m still somewhat skeptical of the business-logic behind The Globe’s decision to launch its paywall-only site last fall alongside its primary news portal, Boston.com. From a revenue standpoint, I can’t see the full-paywall site, BostonGlobe.com, bringing in nearly enough subscription income to compensate for the traffic and ad dollars it will, and to some degree already has, leeched from Boston.com (at last count, in February, BostonGlobe.com had only recruited about 16,000 paid digital subscribers, many of whom had taken advantage of the site’s introductory offer of 99 cents for the first four weeks but may not stick it out past the trial period). It might’ve been wiser for the paper to start out with a metered paywall to warm users to the idea of paying for content before erecting a full-blown ten-foot-tall paywall around its most valuable content under a new, separate domain name. But then again, who really knows – maybe it’s a step in the right direction longterm? I’m not the one having to make the tough calls, so I’m in no position to judge.

At any rate, the segregation of the two sites has given the paper’s parent company the freedom to make BostonGlobe.com a rich, immersive and interactive user experience that few other news organizations can match. Why, you ask? Because of a little trend gaining steam in the development world called “responsive design.” See, BostonGlobe.com isn’t only a standout site because of its sophisticated use of white space, its wholehearted embrace of web fonts and its visual-first approach to story art, but also because of its cross-platform capabilities. No matter what device you view BostonGlobe.com on – desktop, tablet or mobile – the site retains its same slick look and feel. The site’s adaptive technology allows it to detect the screen size of the user, then adjust its layout to fit the exact resolution at hand. On desktops, that means content stretches to fill the entire browser window, and the grid restructures itself appropriately when you decrease or increase the size of the window. This eliminates the need for time-honored design standards like the 960 grid-system, which is based upon the Desktop-centric idea that all users have at least a 960px resolution. Now, the grid can be as big, or as small, as the user wants it to be.

The site’s successful display of a dynamic range of real-time content on any-sized device also essentially eliminates the need for the “app-based” environment that’s been the staple of the iOS5 and Android operating systems in recent years. To test it out, I gave the site a whirl on my iPad, and was pleased to see the front page neatly rearrange itself to fit the new orientation, just as any of the best platform-specific news apps out there do, including The New Yorker and Wired apps for iPad. But just because BostonGlobe.com doesn’t need a native app to display its content beautifully on tablet and mobile devices doesn’t mean that it won’t be missing out on the growing app-centric marketplace for publishers. As Apple and Google continue to centralize digital consumption patterns into the app-based model, BostonGlobe.com may be a little too ambitious in thinking it doesn’t need to play nice with the big-boy tech companies to reach its audience. On the other hand, however, it could prove to be a brilliant move for the paper, setting an example for other publishers and app-based companies in general who want to break free from the often constricting, not to mention pricey, cost of participating in the app marketplace.

For its design alone, BostonGlobe.com deserves any number of awards. Its bold, minimalist interface allows content to shine above all else, free from clutter and distraction. And with high-res horizontal photos, block-quotes, inline video and stylized headlines that grab your attention without hurting your eyes, that content stands out even more. What’s more, the site has the functionality to back up its aesthetics. Its “Save” feature allows users to bookmark articles for reading offline on any device, which, even for sites without responsive design, is a brilliant feature for a news site to implement. Moreover, its “Story Flow” panel at the bottom of each article allows users to click seamlessly through to stories on similar subjects, just as readers would in an old-fashioned newspaper with physical sections. Which leaves me with one nagging question: Is BostonGlobe.com too imitative of physical newspapers to attain success in today’s short attention-span digital audience? Is it too skewmorphic to succeed in an SEO-driven, hyperlinked news economy?

I guess we’ll have to wait and see.

Leave a Comment