Just in time for the 2010 ESRI User Conference, our team has been doing some investigation and testing of several apps running against ArcGIS Server 10 in the Amazon cloud.  What follows is a summary of our experience building and deploying a couple of apps against ESRI’s cloud-based solution.

Background and Business Case

In several presentations, blog posts, and articles of the recent past, Dave and I have detailed some of the Flex API work we have been doing for the USDA Forest Service Forest Health Technology Enterprise Team (FHTET). Historically, the FHTET team has collected and analyzed data describing the effects of major forest pests on the landscape. To this point in time, information has been disseminated by means of an annual pest report in hard copy format. While GIS and hard copy maps play a role in the preparation of this report, as a static report it does not fully leverage the data exploration and analysis tools available within today’s geoweb applications.

The current software development initiative is utilizing ArcGIS Server 10 and the Flex API to create a series of Rich Internet Applications (RIAs) facilitating the distribution of forest health related information to a variety of audiences. In addition, the FHTET team has elected to use this initiative as a test bed to determine the ease and speed with which custom applications can be deployed to the Amazon Web Services (AWS) cloud with ArcGIS 10 for greater scalability and convenience.

Given the richness of the FHTET data and the desire for novel UX elements in the applications, the FHTET organization has elected to pursue a fully custom web implementation based upon ESRI ArcGIS Server 10 and the Flex API with a considerable number of custom widgets and extensions of the DynamicMapServiceLayer. We have based our implementation on our considerable background and experience customizing the ESRI’s Flex Starter Kit to produce a template that now serves as the starting point for many Flex-based applications leveraging ArcGIS Server 10.

The Forest Pest Conditions Viewer

A FHTET Forest Pest Conditions Viewer public application (http://fhtet.dtsagile.com/fhtet/Flex/FPC#), shown below) was first developed to assist public users in exploring the impacts of a number of forest pests for different forest service regions. The number and identities of the pests to be displayed within the Flex application are configurable by each forest service region so that users can see the “top N” pests for a given region based on the decisions and experience of forest health professionals.

Since the application is designed primarily as a data exploration tool, required interaction from the user is minimal. Once a region, state, or county is selected, the application makes a service call to get updated data as JSON and render the results for the user. Region and county selection can be done via the map or via pick lists in the search pane located at page left. In addition to the prominent bar charts rendered across the bottom of the map showing affected acreage and number of counties for each pest, data summaries and links to external information are also provided in the dockable left pane. Included in the tabular data summaries is a function to generate a chart showing pest damage trends for all years in the system and to view information on pests of interest. Functions to generate preformatted pest reports and to export raw data in CSV format are also provided.

The Disturbance Mapper

In addition to the public data explorer, FHTET has deployed a secured Disturbance Mapper Application (Figure 3) designed to use remotely sensed data to detect the presence of pests on the landscape. The application is targeted at individuals performing statewide and regional flight planning for aerial pest surveys. The goal of this application is to optimize flight planning and reduce total survey costs for aerial pest surveys by allowing planners to target areas of interest through map exploration in a web browser.

The basis for identifying areas of interest for pest surveys is based upon change detection data from Modis pre-processed imagery. An number of custom map layer extensions (extending the DynamicMapServiceLayer) have been implemented that allow flight planners to adjust thresholding settings on the change detection imagery to view differences in forest green-up and senescence that signal the presence of tree stressors.

Migrating to the Cloud: As Easy As…

For some time now, the proverbial cat has been out of the bag that at version 10 of the ArcGIS product stack, ESRI will be offering a cloud-based solution for ArcGIS Server. Based in the Amazon Web Services (AWS) cloud, this deployment option provides Amazon Machine Images (AMIs) preloaded with ArcGIS Server for ESRI customers desiring quick deployment, scalability, and flexibility in their GIS infrastructure. So what do we, as architects and developers, need to know to be ready to deploy our custom ArcGIS Server apps to the cloud?

The first thing readers to know is that the process is just plain easy and will simply require a few tweaks from your normal deployment patterns for custom apps built against ArcGIS Server. Figure 4 maps the major system components in a typical example ArcGIS Server solution to major system components used in an ArcGIS Server 10 AWS/cloud implementation.

Once an ArcGIS Server 10 AMI has been launched in the Amazon cloud and sufficient storage space has been purchased and configured, the deployment of an on premise application to the cloud is very straightforward.

· After RDP-ing to the running AMI, the developer simply pulls in (via FTP, copy/paste works for small items) the deployed application and updates any configuration settings

· The RDBMS instance supporting the application, SQL Server in our example, is detached from the on premise deployment database server, copied to the AMI and reattached as a SQL Express instance (we’re well under 4GB).

· In our deployment of this application, we split our geodatabase into operational and base layers. Base layers that do not get edited are stored in a File Geodatabase on the AMI guaranteeing acceptable performance, while operational layers that are editable are stored in an instance of ArcSDE Workgroup on SQL Server Express.

· Finally, any map documents required to support the ArcGIS Server map services are copied to the AMI and the data sources are reset to reflect the new data locations.

It really is just that easy and straightforward.

What about that data thing?

Some readers may be asking themselves why our migration story splits a perfectly good enterprise geodatabase running against SQL Server into a File Geodatabase and Workgroup instance of ArcSDE. The answer lies in that Enterprise Geodatabases are supported by another type of AMI in the cloud, and for our test bed project, another AMI means more money. In addition, the Enterprise Geodatabase AMI is PostgreSQL-based and while the migration process does not involve any magic, it would have required a little more time and effort to get our tabular data in there as well. So we elected to store static layers into a File Geodatabase to guarantee acceptable performance and store editable layers in ArcSDE Workgroup running against SQL Express on our existing AMI, safely under the 4GB file size limit. There are no tile caches used in this test bed deployment.

Conclusions

The cloud-based deployment option available under ArcGIS 10 is sure to present an excellent option to a variety of organizations who have wished for more scalability and flexibility in their existing ArcGIS Server infrastructure. Our experience to date has shown us that for organizations where rapid deployment is critical, ArcGIS Server AMIs can be deployed in approximately 20 minutes (exclusive of data and application loading and configuration times). The ability to create additional AMIs from an already configured instance when coupled with the Amazon Load Balancer means that gaining capacity rapidly when necessary is a real benefit of this new development in the ESRI product stack. This scalability on demand, when viewed against the backdrop of the typical software and hardware procurement process in many organizations is a very real benefit. Furthermore, the flexibility provided to the organization through the capability to deploy this additional capacity on demand, rather than having multiple ArcGIS Servers sit idle awaiting the next emergency response event or natural disaster reinforces this benefit.

If you stumbled across this post and are planning on attending the 2010 Developer Summit in Palm Springs, please consider joining us for an evening of socializing, networking, and good company. You can view the details and RSVP here.  Be sure to provide us a valid email address so we can send you directions as the date approaches.

Dave has proposed talks on:

• Using ArcGIS Server with Ruby on Rails – And why not?! All the kewl kids with their turtlenecks and soul patches are using Ruby and geo-geeks can to.   Dave will give us the rundown on BDD within the Rails framework and cook up some ArcGIS Server SOAP API calls.
• Secure Applications with ArcGIS Server – The result of a project we did awhile back using the JavaScript API building an app to comply with the security requirements of DHS, this presentation will deal with requests over SSL as well as token-based access to ArcGIS Server through a proxy.  Lots of technical goodness in this one…

I have proposed to collaborate with Mike Juniper, another developer in our shop, to talk about building web applications that target mobile devices (specifically iPhone and Android).  Our talk is titled “There’s an App for That: Building iPhone/Android Applcations iwth OpenLayers and ArcGIS Server” and should turn out to be pretty interesting.  At present, very few websites (and even fewer mapping websites) pay attention to mobile device user agents when rendering pages for client consumption.  Did you know there’s a gesture-enabled version of OpenLayers out there right now that can be used in building iPhone and Android specific web apps?  We’ll leverage the OpenLayers client, mobile-specific CloudMade tiles, and ArcGIS Server to build a web app that targets your handset.  Specifically we’ll build a location aware application that allows users to view and create feedback items pertaining to their current location.  If there’s time we’ll try to work in some location based searches, etc.

Dave also eluded to some other nice shiny bits we’re going to try to work in to this presentation.  We’re collaborating with another company and trying to develop a mapping app against their edge-cached, cloud front image service.  The service is access controlled and enables private distributions of map data.  How’s that for a whole list of buzz words for ya?  Once we figure out what all that stuff means, we’re going to write an app against it!

Why vote for our presentation?  Well you’ll be able to bang away at our app from your Android device or iPhone and do your best to embarrass us in front of a room full of people.  As usual we’ll share the source and give you a place to start.  Oh, and did I mention “edge-cached cloud front image service”…sweeeeeeet.

The community-submitted presentation abstracts for the Dev Summit are now posted on-line over here. It’s your conference and ESRI is giving you the opportunity to choose the content.  Since these presentations are all recorded, you don’t have to be attending the Dev Summit to vote – just setup an account (free) and vote for what you want to see – either live or recorded.  Please note that unless you’ve visited the voting site previously, you’ll need to establish a new account and login.  You don’t use your ESRI ID if you already have one of those…registration is quick, easy, and non-invasive.

Fact: One of the most critical components of a highly usable software system is that it be highly performant…in other words, fast.  In the realm of the GeoWeb, this means that applications should load fast initially, and that responses to user activity within the application should be as quick as possible.  We now live in a world of instant gratification when it comes to our news, our hobbies, our finances, and above all our interactions with computers. Users don’t like to be kept waiting for 5 seconds every time they use your app to request information, input data, or perform other common processing tasks.  Gone are the days when watching the blue bar move in a web app, or watching the cursor spin on a wireless app were the status quo.  And frankly with the amount that we, as consultants, get paid to build custom software, the user shouldn’t have to wait.

Headlong into the breach then for our fourth and final lesson…

Lesson 4: Maximize Real and Perceived Performance

In all my years of software development, I’ve never had a user or a client come to me and say,

“Ya’ know Brian, this app is just too fast…please slow it down.”

No matter how much time architects, developers, and UX folks spend designing and building the sexiest GeoWeb app you’ve ever seen, all that work is wasted if the user base is frustrated by long wait times to execute functions within the application.  It doesn’t matter how well you automate a customer’s workflow if they’re left sitting and twiddling their thumbs for 5-10 seconds every time they push a button.

Real Performance

There are a whole host of tools in the IT and software professional’s toolbox that help us maximize application and database performance.  Processing power and RAM, faster network, server farm configurations, database indexing and denormalization, etc., etc.  Those standard tools are beyond the scope of this article as they are relatively well understood.  What I’m most interested in covering is how to squeeze more performance out of our maps given a fixed IT configuration.  Real performance in our mapping realm means configuring map services to be fast from the start, caching whenever possible, and applying novel techniques to handle large datasets.

Fewer Layers, Appropriate Symbology and Annotation

I covered this point in the first post in the usability series back in July.  The idea that users need to see all of the layers in your geodatabase to accomplish a task is ridiculous.  No matter how big your server farm is, no matter how fast your processors are or how much RAM you have, large numbers of map layers are going to slow the system down.  Focus applications on 3-5 operational layers over a base tile cache to reap significance performance gains over the kitchen sink style application paradigm.

Another trend that we frequently see in diagnosing map-related performance issues is a map that contains complex symbology and excessive annotation in the absence of a defined business need.  A dynamic map service showing highway signs over the highways is neat but these symbols need to be rendered in four parts on the server (red, white, blue, and text) and take much longer than standard line types.  In most cases, variance in line weight and color, coupled with scale dependent rendering by road type would convey the same information to the user without the performance hit.  Save the complexity for cached services and be as simple as possible in dynamic map services.

Annotation is a valuable tool for adding information to maps beyond the spatial relationships conveyed by the feature geometries.  It is also frequently misused.  Applications that annotate large numbers of layers with all manner of non-critical information rapidly become cluttered and unusable, in addition to incurring the performance hit associated with rendering all that annotation in the map.  Do parcels really need: PIN, dimensions, area, address, owner name, the names of the owner’s seven children and summaries of the last 5 years tax returns?  Probably not.  Keep the annotation appropriate for the map scale and germane to the information you are trying to convey.  If it isn’t relevant to the business problem at hand…leave it out.

Caching

One of the primary tools in our toolbox for maximizing the performance of GeoWeb style applications is the map tile cache.  Think back to just a few years ago (pre-February 2005) and the thought of using a system of cached tiles in the era of “Web GIS” was unthinkable.  As geo-technologists we were “special” and our data was simply too critical to have any latency.  Our apps had to show that parcel split or that new forest boundary seconds after it came into existence.  I think we can all agree that in February 2005, Google Maps showed us all that this line of thinking was just a bunch of hooey.

Nearly all emerging applications in the GeoWeb realm wind up leveraging a tile cache for base maps within the application.  However in many of the more recent systems we have worked on, we have arrived at the assertion that you should cache as much as possible. With the exception of real time battlefield applications for the military, emergency response applications, and a very few other notable exceptions, we hold that you should cache everything.  It is a relatively simple thing to set up an automated process the update a map cache of dynamic data at a time when everyone has gone home from the office and system utilization is typically low or nonexistent.  Trust us, your users will wait 24 hrs to see that parcel update…really.

So, select a server side technology that supports caching map tiles for representation in the client, use a client side map viewer that will efficiently render cached map tiles, and automate cache creation/refresh if necessary to maximize map performance out of the gate.

Cluster Large Datasets

One of the challenges that we routinely face as developers in the age of the GeoWeb is how to deal with the representation of large numbers of geometries in the client.  Direct rendering of geometries in the client becomes the preferred method of visual representation when we need those geometries and their attributes in the client for speedy response to user action: mouse over balloons, etc.  While Flex and Silverlight offer opportunities to render larger numbers of geometries in the client efficiently, JavaScript based applications begin to experience performance degradation in the client after a couple hundred features or so.   Start dealing with complex polygon geometries and the number is less than that.  And even Silverlight and Flex aren’t going to get you speedy rendering of thousands of geometries.

Consider the application shown above.  This application supports a global surveillance network monitoring avian flu (H5N1) and is built using Microsoft Bing Maps(TM) and copious amounts of JavaScript.  If we attempted to actually render global sample locations on the map, we’d likely crash the client as we’re dealing with approximately 50,000 sample locations.  Instead, we elected to cluster sampling points, aggregating data at small map scales and gradually showing additional sample locations as the user zooms in to larger map scales.  The process looks something like that shown below:

This approach allows us to determine the map extent, select the points that fall within the current map extent, and then apply a clustering algorithm to aggregate points near one another before sending the details back to the browser.  Another key point that we discovered is that spatial isn’t always better.  To perform the above tasks with data stored in ArcSDE, the average compute time was ~5 seconds.  Using SQL Server 2008 Spatial data types natively got us down to about 2 seconds, while simply maintaining sample points as X and Y columns with the rest of the non-spatial data got us to less than 1/3 of a second response time.  Don’t be afraid to leave traditional spatial technologies behind when you can improve performance without sacrificing data integrity!

Using the approach described above, we were able to render points on the map AS FAST as the Bing(TM) tiles come into the browser.

Perceived Performance

Another powerful and underutilized means of improving the user experience within an application is to leverage implementation methods that provide perceived performance gains from the user’s perspective.  Because users seldom, if ever, care about the implementation details that solve their problem, we are able to use a bit of slight-of-hand to make a data intensive application very responsive.

The basic message for tweaking perceived performance is actually very simple: be lean on the wire.  If you are building a data heavy app, and most of the line of business apps we build are, then take small bites rather than trying to ship down everything the user “might” ask for at once.

JSON and GeoJSON have become two popular and related formats for shipping data to the client for further processing and rendering, but if performance is the name of the game, don’t be afraid to depart from standards and use a custom JSON formatter to send only the necessary data down to the client.  In this case custom == fast==good!

Let’s return to our H5N1 application for just a moment and consider what happens when we return all those points to the browser.  Our custom JSON formatter returns only the information necessary to properly render points over the map.  But at the same time, a user expects to see additional detail on sample sites when they mouse over a point so what’s a developer to do?  Sending all information for all points down and storing it in the client is certainly one option, however the odds that a user is going to mouse over all of the hundreds of points typically in any given map extent is miniscule so why send all that data over the wire? As shown in the example immediately above, when the user hovers their mouse over a sample point, we quickly shoot off a tiny XHR call to get a subset of the details.  If they want more, they can click the View Details link in the mouse over window to get more information.

As another example, let’s consider what happens when a user requests all the details for a site.  The site flow dictates that we render the full sampling details in a tree view to the right of the map.  For our purposes we selected the Yahoo! YUI tree view for reasons that will become apparent in just a moment.

When the user requests detailed information on an H5N1 sample site, we go to the server again via XHR call and get only the top level information that you see rendered above.  This is why we selected the YUI tree view over other JS-based tree views.  Dojo, JQuery, etc. either work on a full data store to render information or want to bind to a JSON object that has been fully filled.  YUI lets us load information on a node by node basis and this is critical to maximizing perceived performance in this case because, again, the likelihood of a user clicking on every single bird species in small.  If the user clicks on a species in the tree view, we go fetch only the next node of information, and so on, and so on.  Be lean on the wire.

Parting Shots

So here endeth the homily, go forth and develop some more.  Have I hit all the critical points? Probably not.  The field of usability and UX carries with it a substantial set of principles for buidling killer apps with great interfaces and my periodic usability rants are less a codex and more an appeal to the wider development community to continue to hold web GIS at arms length and continue to the march toward more highly performant, usable, focused applications.  The take away from all this should be that we, as a community, need to focus more effort on providing our user base with relevant information right now, and less effort on flexing our GIS muscles by seeing how much crap we can cram into a browser.

While that statement may sound a little rough (nobody should be out insulting their customers and clients after all) it is not meant as such.  I most certainly am not saying your users are foolish, stupid, or otherwise deficient in any way.  Recall Scott Karp’s assertion that in the age of Google,

“…there are no stupid users, only inadequate designs”

On the contrary, I simply mean that otherwise well-meaning users frequently do things within an application that the application does not expect or that create coding/logic challenges for the developer.  Our task then in building next generation systems to support/integrate with the GeoWeb requires that we anticipate as many corner cases or unexpected results as possible, and architect and code solutions to prevent the user from becoming frustrated at best, and destroying critical information at worst.  Keep the user focused on the application, and put out fires before they arise

With that intro, here’s lesson 3…

Lesson 3: Protect Users from Themselves

So you’ve built your shiny new GeoWeb application, and you’ve done everything right…required field validators, dynamically enabling/disabling input fields to prevent spurious results, targeting distinct workflows to reduce errors introduced by open and undefined function execution, enabling/disabling map tools appropriately.  Your turn the users loose on your application and what happens?  Defects start to roll in.  Well, not really defects so much as comments of frustration and questions on “How do I make this work?”.  Turns out, the application needs more work: user’s have been entering incorrectly formatted dates, and your click function to select a point on a route is nearly unusable.  Explaining to a run-of-the-mill user that a point has no size and a line has no width so selecting a line feature exactly by point is difficult is a fool’s errand.  Prevalidation of inputs, and novel approaches to handling open-ended map interactions (like clicking a point to select a road) are required to prevent users from creating invalid data, removing critical data accidentally, or entering an endless loop of retries when interacting with spatial data.

Validate as Soon as Possible

Ever bought plane tickets from a certain large travel site? The basic workflow is that after the user has gone through the multiple screens to purchase tix, just as the purchase is about to go through, they get a validation message telling them they need to go back to page 2 because the middle initial field is required…or some other ridiculous message that should never occur.  Even better, when the user returns to said page, the offending field may be highlighted, but data in the other 20 required fields was never cached and is now gone.  How fun is that?

Our roadway manager from Lessons 1 and 2 has been such a good an enlightening sport on the subject of usability that we’re going to leverage her experience again here.  Consider the figure below.

Our roadway project manager is updating a road segment to reflect the correct extent of her paving project.  In this case, she’s going to manually enter the beginning point (mile number) of her project on the highlighted segment.  In this case, she indicates that her project starts at mile 90 on Route 40.  Focus on the data entry text box is lost as she goes on to do other work and what happens next is shown below.

Uh-oh!  Let’s look at what happened here.  We have, in the browser, knowledge of the start and end mile markers for our PM’s road segment so we can help her out a little.  The system has done some quick validation and noted that 90 is less than the allowed minimum mile number for this road segment, shown a red ‘X’ which is almost universally understood to mean something has gone awry, and has displayed a message telling her what’s wrong with the data.  In addition, another important thing has taken place.  We have reset the value in the input box to the original value.  We have not destroyed or altered any data, nor have we asked out user to remember what the original value was.  Much like the medical profession, we as developers must “first do no harm” to our client’s data.

And when did all this good validation happen?  Right now!  We didn’t let the user play with the structures grid at the bottom of the page, we didn’t let her make any other changes only to warn her five minutes from now.  We caught the issue early, provided her with a visual cue, preserved the data, and gave her a potential solution to her problem…all with a little JavaScript.

Validate as soon as you possibly can to maintain data integrity and reduce/eliminate user frustration with your app.

Prevalidate Whenever Possible

If validating early is good, prevalidating to avoid problems entirely is even better.  So I know it’s painful but we’re going to leave the realm of the roadway project manager for a second in favor of world health issues.  We’ve built, and continue to enhance, a system for a prominent world wide NPO to track H5N1 (bird flu) incidence around the world.  The system is used by personnel in multiple countries around the world which presents some special issues surrounding data entry…for this specific example let’s consider the entry of dates.

While folks in the US are used to entering date information as MM/DD/YYYY or some variant thereof, other countries like DD/MM/YYYY or YYYY/MM/DD.  Heck some countries (Nepal comes to mind) have entirely different calendars that map by mathematical formula to the Gregorian calendar we’re all familiar with.  While we could simply format date entries to UTC, etc. on the server side, this requires logic at the server to accept information on where in the world the user is entering their data, and complex client logic to turn that UTC into a date that is meaningful to the local user on the other end.  No so fun to deal with.  Why not just use an internationalized date picker?  This is prevalidation at it’s best, the date picker shows a calendar to the user, and when the user picks a date, it is formatted and transmitted in the appropriate format for the local users.  The use of a simple JS widget completely eliminates any chance of an invalid date, or a correct date incorrectly formatted, from squeaking into the system and frustrating or confusing users.  At the same time, the picker knows that a date entered in Europe needs to be formatted differently for a user in the US.

Short Circuit Open or Vague Inputs

So we’re going to jump back to check in on our friendly roadway project manager for our last point in the lesson.  She’s back in our system setting the segments up for a roadway project and telling the system where her paving project is at.  This time, she will be setting up her project road segments by clicking on the map.  This is where things can get interesting because the user can click anywhere she wants.

In the figure immediately above, the user has clicked, whether accidentally or intentionally, a point that is nowhere near the roadway she is working on.  Now, we could try to write code to maximize the efficacy of buffering the point to intersect with the line based on map scale but things can get pretty complex when scale dependent buffers etc. are involved.  A run of the mill user is going to get red faced quickly if we keep telling them to “try again you’re not close enough”.  So why not do something like the result shown in the figure below?

In response to a less than optimal map click for setting the end reference of a roadway project, we simply snap to the closest point on the selected segment.  We’ve avoided coding complex scale dependent buffer/intersect logic that the user will never understand anyway, we’ve avoided the potential for an infinite click-retry loop for the user, and we still get a value that, while it may be incorrect from the user’s perspective, is still a valid point on the segment.  If she doesn’t like it she can enter the mile marker by hand or simply click again.  Low friction protection for the user and it short circuits a whole series of potential outcomes, all of which are bad.

Up Next…

Since I’ve yet to meet a client or user that says their system is too fast, our next usability topic(s) will deal with both real and perceived performance in your GeoWeb application and how to maximize each of these critical performance factors. Stay tuned…

Part 1 discussed hiding unnecessary complexity from the user, while our lesson plan this time around uses extensive feedback mechanisms within a site to keep the user on our side and make them feel comfortable when visiting our site.  A confident user is a comfortable user. A comfortable user gets what they need and gets back out fishing, golfing, spending time with their kids, or otherwise doing something else besides staring blankly at your interface.

With that, here’s lesson 2…

Lesson 2: Provide Feedback to the User

Nothing presents a bigger usability hurdle than a mapping application that leaves the user wondering, “Well this looks cool, but what do I do?” This is the principal downfall of applications that try to shove GIS into a browser.

Geodevelopers know exactly what to do with three different toolbars and four menus containing all manner of map navigation, query, buffer, and analysis tools but open workflow applications with no guidance are intimidating to line of business users and the public. The web mapping industry at large must learn to develop applications that satisfy specific workflows and to lead users through those workflows within the application with visual cues and feedback in the UI.

The roadway manager introduced in Lesson 1, uses the interface shown above to specify the begin and end points of a roadway project. First, note the simple informational panels that appear in the right hand column of the site to explain how to use the features on this page. Second, accept that a GIS professional knows that when they click a pencil icon, they expect to draw something on a map, but a roadway manager does not necessarily connect these two actions. Thus when the user clicks the pencil icon to set the begin or end point, information appears below the map telling our roadway manager what to do next. “Click the map to set the begin point.” It’s as simple as showing or hiding a <div /> element in the page, and it is a critical usability feature that is often overlooked.

Hold the User’s Hand

I’ve recently read a couple of typical user workflows for web mapping applications that read something like this:

First of all, in my book this is actually two workflows but that’s beside the point.  What really grinds my gears is that when you look at the application built to satisfy this workflow, the architect/developer elected to provide a separate tool to do each of these actions so that the user is free to wander around the site and do things in any order they please.  Not only does the user need to know which tools to pick from, they need to know the correct order.  While eventually the user will figure out the “happy path”, that’s a lot of moving around the app and a big learning curve that is just going to frustrate and turn the user off.  If we know the user is going to perform these actions to get information each time they visit, why don’t we just write a widget that does exactly that???

Now have a gander at the app shown above.  The workflow that this particular application functionality supports is as follows:

To support this workflow we could build a drawing tool with a data entry interface, an a generic intersect tool, a TOC, and a reporting tool, but why in the world would we do that? This is an emergency response application which means we don’t want our user’s hunting and pecking for tools when they could be busy making bad things stop…immediately.  The entire workflow is encapsulated in a single dojo dijit.

1. When the user activates the dijit by clicking on a menu item, they are prompted immediately to draw a shape and are informed on how to close the shape/stop drawing…a double click.
2. Once the user closes the shape, the data entry interface pops up and says “Hey, I need some data!”.
3. As each field is filled in, the form keeps squawking for more data until it has what it needs.
4. Since we already know the user is going to want to generate a report after we do our intersect and save all this data in the database, we ask them which one they want from the get go.
5. On ‘OK’ the system goes away, saves the polygon and attributes, does a bunch of processing steps and pops up the needed report.

We have held the user’s hand, led them through a well defined workflow, and given them the information they needed without all the noise and distraction of 2 menus, 3 buttons, a cup of coffee and two trips to the bathroom.

Reassure the User

When a GIS Professional executes an attribute query or runs an intersect operation, they typically understand that sometimes, the result set is empty. The public, or some other non-GIS user, gets very nervous when they perform an action and are either presented with an empty UI, or are otherwise redirected to something they didn’t expect. Did they delete something important? Is the request still processing? Did the site crash? “What the heck happened to my data?!?”

The importance of reassuring the user anytime something out of the ordinary happens cannot be over stated. Continuing with the roadway manager example, the figure above illustrates a case where the user has selected a project with no location information. Rather than showing the user an empty interface, leaving her to worry about what has happened, the map interface is zoomed to the general area of the project (known from tabular information) and an informational dialog explains that there is no need to worry and provides instructions on how to remedy the missing data issue. There is no need for a complicated exception or null case scenario; a simple modal dialog addresses the usability issue and keeps the user on the right track.

Up Next

Stay tuned for part 3 in the series dealing with strategies for protecting the user from themselves.

It is my sincere hope that we, as a community, are finally moving on from the era of exposing buckets of complex GIS functionalities in the browser.  As noted in a previous post, the GeoWeb is, in essence, all things Web 2.0 writ large on a map. For ESRI customers it is REST, JavaScript, Flex, and Silverlight APIs for ArcGIS Server.  Beyond the ESRI realm it is Microsoft Bing Maps, Google Maps, Google Earth, or myriad FOSS offerings.

Since internet users now have a myriad of choices in where they go for information, we as professionals should be designing highly usable systems that give users relevant information…and give it to them right now. If we don’t, they’ll simply go somewhere else. What this means for all of us as architects, and developers, and project managers in the mapping industry, is that usability trumps features. Since our customers are foresters, real estate agents, state troopers, and roadway managers, not GIS professionals, we need to hide GIS complexity, provide determinate, task oriented interfaces, and answer the user’s question with a minimum of friction and interaction.

While quick performance and a killer site skin with an open, uncluttered layout are certainly important in the age of the GeoWeb, they are only part of the equation. Based on my unscientific observations, geodevelopers are still challenged on the usability front in that the typical application with buckets of data and loads of tools with an unconstrained workflow is still making it out the door and into the market in many cases. Creating great apps for public facing or line of business sites serving non-GIS professionals requires a lesson plan that focuses on the user and on the mental model of how the user interacts with functionality that the geodeveloper exposes.

But I need a full featured GIS in my browser

Oh no you don’t.  We at DTSAgile maintain that there are very few situations that actually call for a web-based GIS. When these web applications are built, they are so complex that only GIS professionals understand how to use them. Perversely, the performance and technical limitations of web development make these applications too limited to be useful for GIS professionals! Give GIS professionals access to professional GIS tools – desktop GIS applications. Citrix or Terminal Services technologies provide an excellent means to do this in a distributed environment. This allows all the GIS applications and data to be co-located in a single data-center, while still having a “desktop” experience at remote locations. Having designed, architected and developed several large implementations, we’ve seen how it can deliver powerful desktop GIS functionality across an enterprise very cost effectively.

If however your goal is to service the public or users in a specific line of business, then you would do well in creating focused applications which help a user solve a specific problem easily – which is exactly what GeoWeb style applications do. The next generation of spatial applications is now arriving on the scene and starting to leverage real GIS analytical capabilities behind the scenes. While the tendency to mimic the indeterminate workflows of desktop GIS packages in a browser has noble goals, implementation, performance, and usability issues usually run rampant and completely swamp any cool-factor. Instead, focus on solving specific business problems by building tools that are tailored to the actual end user.

With that introduction…here’s lesson 1.

Lesson 1: Stash complexity away

Foresters know trees, state troopers know law enforcement, the County Auditor knows real estate assessment, the public knows a whole variety of things, but likely none of these user groups knows specifically what a buffer, intersect, union, or thiessen polygon is. When a roadway project manager asks for all structures near her project, without knowing it she really means,

“Locate point features in the Structures layer that fall within 1 mile of the section of Route 6A between mile posts 12 and 25.”

A GIS professional would know that getting this information requires an initial point selection, followed by a buffer, an intersection with a second layer (roads), followed by a buffer of the resulting road segment, followed by an intersection of the second buffer with the structures layer. The roadway project manager does not know this, nor should she.  Furthermore, why in name of Pete would we give her 5 or 6 separate tools and ask her to figure out that process in the correct order?

Consider the application shown in the figure above.  For those keeping track, the app is an ASP.NET MVC implementation consuming data via the WMS capabilities of ArcGIS Server 9.2 within the OpenLayers client interface.

Fewer Layers

Note how few layers are in the map.  I’m sorry to say so, but if you think your users need or want to see all 150 layers in your Geodatabase, you are dead wrong.  Unless your application operates on or depends on that really detailed point layer of GPS’d mailbox locations for correct function, leave it at home and let’s keep it simple okay?

Consider using a base tile cache with some sexy cartography, and then roll in 3-5 operational layers that contain only the information that the user needs to accomplish their goal.  The example above contains roadways, separated into scale appropriate layers for display purposes, and counties to provide locational context.  To our roadway manager, everything else is noise in the system and doesn’t contribute substantially to her ability to do her job so it is left out.  Using fewer layers also has the side effect of dramatically increasing map rendering performance…which is topic for a separate post in this series dealing with real and perceived performance.

Hide the Details

Check out the minimalist map navigation at the top left of the map and the conspicuous lack of multiple tool buttons, menus, legends, layer lists, etc. in our example above. This application is used by State DOT roadway project managers and all these folks need to do their job is to:

• Specify what road segment a project is on
• List structures along the road (culverts, mast arms, etc.) impacted by a project

Highly useable systems hide complex GIS operations from the user and get the desired answer quickly. The selection, buffers, and intersections that get the roadway manager the information she needs to do her job are hidden inconspicuously behind the “Search for Structures” button highlighted below.

Once the project road segment is selected, this search button becomes enabled, and a single click returns a list of affected structures to the user in approximately 0.3 seconds in this application allowing her to get back to what’s really important…spending that stimulus check.

Future Zen

Keep checking back here for additional posts on usability, performance, etc. for building the next generation of web mapping applications.  Future posts will deal with feedback to the user, user reassurance, protecting users from negative outcomes in your app, real and perceived performance, and anything else I can think of that irks me when I look at some of the “Web GIS” sites I see rolling out onto the internet.

The Geospatial Web, the current darling of location based technologies and neogeography, has been variously described as:

• web mapping – the generation and publication of highly performant web applications that include a map
• mashups – combining spatial data with abstract information to produce novel representations
• a distributed GIS for the web – essentially the combination of elements of traditional GIS technologies with advanced web development tools
• equivalent or synonymous with relatively new technologies such as Google Earth, Microsoft Bing Maps, Yahoo Maps, etc.
• communal or user-generated geospatial content

While each of these descriptions is both incomplete and an oversimplification, I assert that at its core the GeoWeb is a technical foundation of information services and collaborative tools built upon or merged with data that provide a spatial context…a location.  It is indeed shiny “slippy” maps, Web 2.0 application layouts, nano-formats, mashups, LBS, service-based spatial information sharing, but it is also the hardware, software, information architecture, advanced web application techniques, and people who come together to make such an organic endeavor possible.

The number of organizations working with GeoWeb technologies is already large and growing daily.  It seems only logical then, that as a community we engage in an open debate about how exactly all of these databases, services, clients, and functionalities are ultimately going to communicate, integrate, and grow. My endeavor, in the context of this post and in preparation for an architecture panel at the GeoWeb 2009 conference, is to espouse Representational State Transfer (REST) as a standard architectural pattern for the GeoWeb. I’ll try to keep my points brief here with hope of stimulating discussion that will continue at the GeoWeb conference in Vancouver at the end of July.

REST is Best #1: Adopt the architecture of the medium

Continuing my trend of oversimplication, the GeoWeb is simply the web + geo right?  While it is indeed more than just web mapping, at it’s core the GeoWeb commonly leverages the World Wide Web for information access and representation in a client application.  The web is the poster child for RESTful architectures as articulated in Roy Fielding’s doctoral dissertation.  So the pragmatist in me says that if we’re going to build a network of loosely coupled systems to support the Geospatial Web and we’re going to use the web to provide access, why wouldn’t we leverage the architecture of our good friend “dubya, dubya, dubya”?  GET, PUT, POST, and DELETE.  The semantics are well defined and well understood (okay, POST can be a little wonky due to issues of idempotency but I think my point holds).

Every single application exposes its resources through exactly the same interface…the HTTP specification.  Would you, as a developer of a service or a client rather work with 4 HTTP verbs, or would you rather need to discover, proxy, and understand myriad SOAP contracts?  Written a WSDL lately? Read one? I thought so…

REST is Best #2:  It’s about resources

REST-based services (or REST resources for my “high-REST” friends in the audience) are true “web services” in every sense of the term since they’re based entirely on HTTP and URIs.  Map images of downtown Las Vegas, your running route, drill pad locations, the location of the nearest Starbucks, parcel boundaries, a street level image of some guy mooning the Google van; all just pieces of information when it comes right down to it.  What we store, exchange, represent, draw on a map, and recombine into novel information combinations are simply resources…and that’s what REST is all about.  Why make it any more complex than it has to be.  Why would I POST  to a SOAP method or consume an event from a service agent when all I want to know is http://www.myTranspoSite.com/projects/532. (In English: “Get me the location, status, and funding information of the stimulus transportation project going on at the end of my street.”) 

“But what about my complex geoprocessing operation,” you say?  Well, spatial statistics or complex operations that produce a result (buffer, linear referencing operations, etc.) are all fine and dandy for us technologists, but the GeoWeb isn’t just for GIS professionals anymore.  Why expose your client developer, or worse a run of the mill user, to such complexities.  A user doesn’t know what a buffer is and neither does the Dojo ninja that’s consuming your resourcein his JavaScript client…they just want to know how many crimes where committed in their neighborhood in the last 6 months.  Hide complexity within the REST resource and give a simple answer to a complex question.

In an ROA effectively adhering to a RESTful architecture, all I need to know to get the information I want is a URI and an action.  Do I have to negotiate a proxy?  Am I getting cached information? Is there a gateway or firewall in the way?  How do I negotiate that P2P thingy again? Do I need to do a buffer followed by an intersect followed by a 12th level of hell whatchamacallit? Who cares?  Just give me the information I want so I can go play golf, spend time with my kids, wax the car, take a nap, or do any number of other things I’d rather be doing.

REST is Best #3: Serving many masters

The beauty of REST is that you get the standard interface provided by HTTP, with no limitation on data representation.  Why is this important for the Geospatial Web?  Because browsers like HTML and JSON. Because machines like XML, binary, or code on demand.  Humans like pretty pictures and PDFs. And sometime down the line machines and browsers and humans may all change their minds and like some other format.  That’s right folks, the GeoWeb ain’t just for browsers anymore and that means we as architects and developers need to keep multiple resource representations foremost in our minds.  REST is absolutely critical as an architectural pattern for the GeoWeb because it provides a low friction way for us to provide information formats that better match consuming clients and performance requirements.  And as new content types come to light and get defined, REST lets you add support for these novel formats without impacting support for existing/older content types. 

REST is Best #4: Now where did I put that..?

So if the GeoWeb is about publishing, consuming, transforming, sharing, visualizing data with a spatial component, it stands to reason that we need a mechanism by which to discover said information.  And we’d probably also like a way to get back to the information once we’ve been there once.  Or send it to our boss, or forward it to grandma, or shoot it out the list-serv that we’re on.

In short, REST makes the information driving the GeoWeb portable. Once I find a resource I like I can bookmark it to return later, I can throw it in an email and share it with others, I can embed it in my application to be consumed and acted upon in novel ways.  It also means that we don’t need a separate discovery mechanism for our resources because resources can be embedded as content in other resources.   A country resource can provide information on state resources which can provide information on county resources, etc.  Imagine that; an entire network of information and functional resources interconnected by URIs…huh.

REST is Best #5: Performance and Scalability

For most RESTful services (I know, I know…resources) the majority of requests to the resource are probably GET requests.  Do I have numbers to back that up?  Of course not…I’m an architect.  But the logic of the assertion is that presumably many more people are allowed to read/view information than are allowed to create, update, or delete information.  Even in scenarios with lots of resource editors, the GET is still likely to far exceed PUT, POST, and DELETE request counts.  One of the central tenets of the REST architectural style is that whenever possible, information should be cached.  REST gets it’s performance and scalability from this cache-ability, coupled with statelessness.

What this means for the GeoWeb is that we can cache GET requests against a resource.  Thus if I visit a resource URL and ask for information on a highway project, and 2 seconds later my neighbor asks for information on the same project…my GET results will be cached so that my neighbor’s request does not incur the penalty of a database hit, business logic enforcement, object formatting, etc.  Does it matter for a system with 100 users on appropriately sized servers? Probably not.  Does it matter for a large interconnected network of resources with hundreds of thousands or millions of users?  You betcha’.

Statelessness means that a server hosting data or functionalities for the GeoWeb should service requests in a manner that does not depend on context or state between requests or between services.  If you say to a REST resource, “Give me a map of all preschools within 5 miles of my home”.  You cannot simply make a follow up requests that says “Now buffer them”.  The statelessness of the resource on the server means it no longer remembers anything about the 20 preschool points it just handed you as JSON…in fact, the server doesn’t even know who you are anymore and wants your credentials again.  Why is this good?  Because RESTful architectures don’t typically have to contend with maintenance of session state, a server can be swapped out with another in the event of a failure, a sys admin can decide to add three new servers to a farm on Wednesday without existing clients knowing any different, or a group of servers can be allocated to service different types of requests.

Caching and statelessness = performance and scalability all in one neat little package.  Sort of a big deal for people building LOB or otherwise critical applications for the GeoWeb dontchathink?

Parting Shots

Did I miss something in my initial hypothesis that REST is the critical architectural style for the GeoWeb?  Probably.  Did I get something wrong or use semantics likely to raise the hackles of true, dyed-in-the-wool RESTafarians?  Probably…I tend to be more pragmatism than dogma anyway.  Consider joining us for GeoWeb 2009 July 27-31 and see if I can avoid making a complete fool out myself whilst fighting the good fight to make RESTful architectures an integrated part of the evolving GeoWeb. Find a glaring hole in my argument or something I missed?  Leave a comment here so I can study up before doing battle with the SOA/SOAP and P2P/Event Driven folks in July.

Since I’m a botanist by training, and not a classically trained programmer by any stretch, I don’t typically look at a new technology and say “Gee Whiz, this looks an awful lot like <pick your poison> that I learned back in University”.  I’ve got sort of a free form learning process that works for me, and usually get’s me up and running and useful in a relatively short period of time…a matter of days if I have examples and a pattern rather than weeks.  And after doing this for 10 years or so, it seems that you still can teach an old dog new tricks.

The cone of silence

The learning process, especially at age 35, does not benefit from distractions.  I still can’t believe I could sit on my couch and watch a movie while studying in college or camp out in a noisy coffee shop and write portions of my thesis in graduate school.  Now, it’s less about the TV or background noise and more about email barrages, social networks, and the new, new thing on YouTube.  Perhaps the single worst feature of my Twitter client (TweetDeck) is that it is constantly squawking when new updates roll in.  When I need to learn something new from the ground up, the cone of silence get’s invoked. 

This doesn’t mean I don’t talk to my coworkers or answer my wife’s calls…quite the contrary, I’m usually asking coworkers plenty of questions and since my son has a health condition I always take calls from the wife.  What it does mean is that I shut down my Twitter client, fire off a courtesy email to the rest of the company (and stop answering email), and buckle down for a day, or two days, or three if that’s what it takes.  If a coworker is physically on fire and I have the only fire extinguisher, I respond.  If not, then it’s going to be a day or two before I can get to that proposal you want me to work on.  There are no ad hoc requests allowed from outside the office I physically sit in.  Removing all extraneous distraction keeps me focused on the task at hand and allows me the deep dive that I require to wrap my head around something new.  This cone of silence is also an excellent tool when up against a deadline on any project.

Find room to spread out

I can’t learn a new technology at home after work.  I can read a book about it and make some notes and commit some of it to memory, but I can’t kick back, spread out, and mentally digest things.  Wife, two dogs, cat, four year old…you get the picture.  There’s a lot going on at home and working there usually means a single monitor machine or the netbook, and not a lot of room to make a mess.  Learning something new is when our developer rigs at DTSAgile really shine:  Comfy chair for exercising the typical bad posture of a developer;  Big desks with plenty of room for scribbling things down in a notebook or two; Three monitors to spread out your IDE, Google, help system or API documentation, sample databases, etc., etc.; Additional desk space reserved from the latest offerings from O’Reilly or Wrox; Real estate for half drunk coffee from yesterday as well as today’s lukewarm addition to the collection.

I need room to roll up my sleeves and work.  Having multiple information sources in front of my face at the same time speeds the learning process for me and is simply more time efficient.  I’ve gotten to the point now where it’s actually an annoyance to have to hunt and peck on the Windows taskbar  to find that great Google search result or pull up one of 5 sample VS.NET solutions I’ve got going.  Call it ADD, call it information overload, it works for me…but it only works if I’ve got room to spread out, both in the digital world, and the physical one. If I’m truly going to leverage the wealth of information that’s available, I certainly need a big palette to work from, and an even bigger canvas to spread it on.

Know where you’re going

It helps immensely to have an example of how things ought to look when I’m done writing code.  This doesn’t necessarily mean the user experience manifested in the UI or in the behavior of the service boundary, but rather the appearance and organization of the code. Knowing enough to write a dojo.subscribe() is one thing, but where is the best location to put that code given the context in which it will run? In a *.js file for a widget? In the page itself? In some generic widget controller?  Dojo can be used to simply add a few shiny bits to a web client, but it is frequently used as a fully object oriented client application framework, making good patterns and practices very important. 

Having reference implementation code, either from a previous project my team has done, or from another source is usually critical.  If the technology I’m learning is fresh in the market, then I’m happy to give things a little more thought and blaze new trails, but if someone has gone down the path before me, then I definitely want a fully wired example to work from.  Fortunately in this case, my coworkers had been doing Dojo development for awhile so I had a couple of good solutions to work from.  Cheating? Lazy? Maybe, but why in the world would I not try to benefit from the previous experiences of others.  It’s just good sense to know where you’re going…that way at least you’ll know when you’ve arrived.

Breaking stuff

I’m a tinkerer, and have been since childhood.  What good is a perfectly working model, RC car, or a piece of software if you haven’t taken it apart, broken it, and put it back together again?  One of my best resources during the learning process on a new technology is my tendency to tinker, pull things apart, and see what makes them work.  Sure you could read about about the difference between dojo.hitch() and dojo.connect(), but it is infinitely more revealing to test drive both methods to see what works and what doesn’t.

My shop is in the consulting business, which means we need to be experts in lots of different technologies and we need to achieve that level of expertise quickly as technology evolves. The long and short of it, is that I learn best when I’ve gotten to the point of a working piece of software…and then go one step further.  I have found over the years that the difference between kinda’ getting something, and really grok-ing what’s going on, lies in my tendency to want to break stuff and then put it back together so that I know what all of the component pieces and parts are doing.  In the long run, this generally increases my code quality and readability because not only have I taken the time to learn what to do with a given technology, but I’ve also learned what not to do.

References and tutorials

I don’t think it’s too much of a stretch to say that for most developers these days, Google is the most valuable reference book on their desktop.  However, those of you who have already jumped into the deep end of the Dojo swimming pool probably realize that:

1. The API is not particularly well documented with comprehensive explanations and examples when compared to other APIs (JQuery, Prototype, etc.)
2. There are two chief resources online for Dojo:  The Dojo Campus (www.dojocampus.org) and the Dojo Toolkit Site (www.dojotoolkit.org), neither of which sits on a particularly speedy server or offers a comprehensive how-to library.

All Dojo peculiarities aside, even if I was learning ASP.NET MVC, Entity Framework, PERL, Python, Ruby or any number of other well documented technologies or APIs, I certainly wouldn’t rely entirely on what online resources could teach me. 

Perhaps it’s because I was raised at a time when you still went to the local bookstore to find a good read, or perhaps it is because of the countless hours I spent working in my grandmother’s small town bookstore, I still like to have reference books around.  Having two or three books on the technologies I’m actively working in means I can take a book to lunch and read up on a particularly hairy topic, I can look long and hard and dissect an example without blinding myself staring at the LCD, or I can take a volume home and read before nodding off at night.  Absorbing information from multiple sources can only help the cause of learning new technology, and for me the thumbing of pages will always hold something that a scrolling a mouse wheel never will.

Conclusions

So there it is…you can in fact teach an old dog new tricks.  While my learning process is probably not as organized and logical as that of others, it does work for me, and has allowed me to keep pace with what is turning out to be a rapidly changing technological landscape.  What’s next?  Well, the cone of silence is off so there’s probably a couple of proposals in my near future, but the Dojo work is nearing completion, it works well, and I can see Silverlight 3, Entity Framework, ADO.NET Data Services, and probably some Flex in my near future.  I think I need a cup of coffee and a fourth monitor.