Why your back-end tools should be sexy, too

If your internal tools aren’t the same quality and sexiness as your client-facing tools, then your employees aren’t going to be as excited as they could be and won’t be selling your company as well as they could.

While working at Bookr, our design motto was “dead simple; dead sexy.” It’s something that I think is a great, simple goal that most everyone in the company can eat least target. We applied this to our product and customers loved the way it looked. I’ve carried that through (internally) for Blueprint, although I don’t think that “sexy” quite fits our brand here as it did for Bookr.

On almost every project I’ve worked on, we put a lot of effort on client-facing parts of the product and tools that are used internally (often called “admin” screens) just don’t get the same treatment.

Have you ever called up a company on the phone and had to wait while the representative has to navigate an extremely complex system or set of systems to get your information? Ok, have you ever called a company at not had that experience? Doesn’t that frustrate you as a customer? Quite often, I’ve had representatives apologize for the poor quality or capabilities of their software. As a software developer, I don’t want this ever to happen with my products.

So in the last two companies, I’ve been working to promote as much, or nearly as much, effort on the design and thoughtfulness and usefulness of “admin” screens as I do on the client facing stuff.

For Blueprint, not only are our clients using the product directly, but account managers and analysts use the product for consultative services. They get the benefit of client-facing tools, but when they cross into the “admin” parts of the site I don’t want them to feel put off.  We are constantly looking at the tools we have built for managing, tracking and maintaining client accounts and client information — tools that clients don’t have access to — and they way we use those tools to try to make sure they can be simple and sexy.

Advertisement

Emotional search with Twitter

I’ve been reading the Twitter API docs and just discovered that Twitter lets you search based on the tone of the tweet using emoticons. For example, type this search:

#829strike :)

Twitter returns tweets about the Fast Food strike on 8/29  with a positive attitude.

If you want to see the other side of the the emotions, use a frowny face.

#829strike :(

This works on Twitter Search as well as through the API.

Using emoticons as operators to represent attitude is pretty cool.

We fixed rails redis store to rapidly expire 50,000 keys daily from a 3.5M key cache

In building out Blueprint, we have taken an approach of “cache everything” (or what we think is most important) to make the app super responsive. Our customers love this.

What we actually cache is, essentially, a data digest used in the reporting view (so not the view or the model per-se). These digests could be a daily, weekly, monthly or annual view of a perspective on a data cube. Each day we add new, updated data to our cubes and need to expire the cached items that pertain to that day (e.g. latest week, month, year). In addition, we’re often recalculating categorizations for clients which means expiring their entire cached history and rebuilding it.

And so, we will expire 50,000 cache keys each morning as we push in new data (and then we go rebuild them). For a given site we might delete 1000-2000 keys.

Up until this week we used the delete_matched method to remove all of the site’s keys in a single call (optimizing our redis time). However, delete_matched relies in the redis keys method which is slow, blocks the redis server and isn’t meant to be used in production. In fact, I could watch our redis server’s CPU go from 0.1% to 100.0% exactly during the call to expiring site caches. Because redis is mostly single threaded, pinning the CPU for processor-heavy operations holds back all the fast GET calls from our cache reads and slows the site to a crawl.

The solution was clearly we couldn’t rely on redis to tell us what keys were in the cache. Michael and I whiteboarded this and built (well, Michael mostly built) a replacement cache store that inherits from RedisStore and effectively does three changes: stores the key name into a set (we partition sets, which is hidden in the code below) when the cache entry is written, remove the key name from the set when the cache entry is deleted, use the set to find the matched keys for delete_matched and remove those from the cache and the set.

      def delete_matched(pattern)
        sub_set = find_matching_keys(@set, pattern)
        Redis.current.srem(@set, sub_set)
        Redis.current.del(sub_set)
      end

      private

      def write_entry(key, entry)
        Redis.current.sadd(@set, key)
        super
      end

      def delete_entry(key)
        Redis.current.srem(@set, key)
        super
      end

I was pretty surprised not to find other solutions for this already or that the redis-store implementation doesn’t do this automatically. However, when we implemented it, the tricks in partitioning the set (to keep that read time low) lead me to believe it’s not a generalized solution (although you could use hashing to generalize it). We also had to build a process to build up the set from the existing 3,541,239 keys in our cache, so we’ll see if that takes the weekend or just the night.

Nginx maintenance page configuration with load-balancer health check

We wanted to add a maintenance page to our rails app running on nginx. Capistrano provides a simple task to show your maintenance page cap deploy:web:disable. It includes basic instructions for how to configure nginx, although I found these didn’t quite work for us as documented.

One thing the configuration suggests it to return a 503 status code so that crawlers know the site is temporarily down and will retry the crawl later. This made sense, but when we did this in front of the load balancer it automatically sent it’s own 503 to the browser because it detected that the app server was returning 503 (which is what it should do, or roll over to another server). What we needed was the ability to have our load balancer run it’s health check on the app server without getting the 503.

While I have read that that “If is evil” in nginx configurations, I found that I could check whether the request was the load balancer and return a 200 code within the maintenance handler:

        if ($request_filename ~ '/health_check') {
          return 200;
        }

I also learned a lot from this older post (like how to allow inclusion of graphic assets such as our logo). But the key trick was to put a bare ‘=’ on the error_pages line so that Nginx would return the handler’s response code, rather than 503 for all pages.

      error_page 503 = @maintenance;

Here is the entire configuration that we used (put this at the server level).

      recursive_error_pages on;
      
      if (-f $document_root/system/maintenance.html) {
        return 503;
      } 

      error_page 503 = @maintenance;
      location @maintenance {
        error_page 405 /system/maintenance.html;

        if ($request_filename ~ '/health_check') {
          return 200;
        }

        if (-f $request_filename) {
          break;
        }

        rewrite  ^(.*)$  /system/maintenance.html break;
      }

Sorting addresses treating numbers numerically

I was working on a small sorting task where we wanted to sort entries like these:

1 Sesame St
2 Sesame St
12 Sesame St
Building A
Building Z
Suite 1
Suite 2
Suite 100

As you can see this probably is what you’d expect — the numerical parts of the strings are sorted numerically. But standard string and number sorts don’t do that. So I worked out a quick compare routine for using in sorting and thoughts I’d share it.

        public static int Compare(string a, string b)
        {
            var numberA = ""; 
            var numberB = "";
            int cmp, ap = 0, bp = 0;
            char ca, cb;

            while (ap < a.Length && bp < b.Length)
            {
                ca = a[ap];
                cb = b[bp];

                bool aDigit = Char.IsDigit(ca);
                bool bDigit = Char.IsDigit(cb);

                if (aDigit && bDigit)
                {
                    // Create a number from all contiguous digits in string a
                    do
                    {
                        numberA += ca;
                        ap++;

                    } while (ap < a.Length && Char.IsDigit(ca = a[ap]));

                    // Same for string b
                    do
                    {
                        numberB += cb;
                        bp++;

                    } while (bp < b.Length && Char.IsDigit(cb = b[bp]));

                    // Compare the numbers numerically
                    cmp = Int32.Parse(numberA) - Int32.Parse(numberB);
                    if (cmp != 0)
                    {
                        return cmp;
                    }
                    numberA = numberB = "";
                
                    // If they match look at the characters following them
                    cmp = ca - cb;
                    if (cmp != 0)
                    {
                        return cmp;
                    }
                    ap++;
                    bp++;
                }

                else if (aDigit)
                {
                    return -1;
                }
                else if (bDigit)
                {
                    return 1;
                }
                else
                {
                    cmp = ca - cb;
                    if (cmp != 0)
                    {
                        return cmp;
                    }
                    ap++;
                    bp++;    
                }

            }

            return a.Length - b.Length;
        }

I have to apologize that there’s no test code for this. But I hope it’s helpful to some. Should be easy enough to translate to other languages.

CoffeeScript makes Jasmine so much better

At Bookr we are moving our JavaScript to CoffeeScript mostly with the intent of making things more readable. I took some time out this weekend to familiarize myself with it by working through our code and doing some of the conversions. I’m not convinced that everything is more readable (example below) but it does make our Jasmine tests much easier to read.

Continue reading “CoffeeScript makes Jasmine so much better”

Customizing the Android EditText behavior with Spans

Friendly names for contacts in the EditText field of PhotoRocket for Android app
Contacts show as friendly names and are underlined

While working with PhotoRocket, I designed a different edit experience for their Android app to treat contacts similarly to the other PhotoRocket client apps. Specifically, I wanted the EditText control to render contacts using their “friendly name” and to treat those as a single entity for navigation and delete.

As you can see in the image at right we chose to underline the friendly names which provides a visual indicator that they are a unified entity and are different than typed in text. This, as well as handling movement events and delete events around the ‘entities’ was possible because of Spans — a feature that allows any object to be attached to points in a CharSequence.

This is a fairly lengthy post, so I’ve divided it into sections on rendering, movement events and handling deletes.

Spannables and rendering

All of our contact data objects implement an abstract class called Recipient that includes basic email and friendly-name data.

public class Recipient {

    private String name;
    private String email;

    public String getName() {
        return name;
    }

    public String getEmail() {
        return email;
    }

    public void setName(String name) {
        this.name = name;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    @Override
    public String toString() {
        return String.format("\"%1$s\" <%2$s>", name, email);
    }
}

EditText fields have Editable content which implements Spannable. The edit control in the form is a MultiAutoCompleteTextView which requires an Adapter to provide and filter the data and place it in the EditText control. Using the filter was the perfect place to convert selected text into a Spannable and add it to the EditText field. So I overrode the filter’s convertResultsToString method to have it handle Recipient items in the result list specially.

        @Override
        public CharSequence convertResultToString(Object resultValue) {
            if (resultValue instanceof Recipient) {
                return ((Recipient) resultValue).toCharSequence();
            }
            return super.convertResultToString(resultValue);
        }

And then added special handling to the Recipient‘s toCharSequence method that inserts the spans.

    public CharSequence toCharSequence() {
        String name = getName();
        SpannableString spannable = new SpannableString(name);
        int length = spannable.length();
        if (length > 0) {
            spannable.setSpan(
                    new RecipientSpan(this),
                    0,
                    length,
                    Spannable.SPAN_EXCLUSIVE_EXCLUSIVE
            );
        }
        return spannable;
    }

What this does is to create a SpannableString using the text of the recipient’s friendly name, then add a span from the beginning to the end of the returned string that includes a reference to the Recipient object. So with the RecipientSpan class I am able to attach a Recipient data object to any part of the text. (The MultiAutoCompleteTextView handles appending this to the entire Editable string in the EditText on the form.)

    public static class RecipientSpan extends ClickableSpan {
        private final Recipient recipient;

        public RecipientSpan(Recipient recipient) {
            super();
            this.recipient = recipient;
        }

        @Override
        public void updateDrawState(TextPaint ds) {
            ds.setUnderlineText(true);
        }

        @Override
        public void onClick(View view) {
        }
    }

Spans are handled specially within the framework, so this design depends on the framework authors not changing how ClickableSpans are used. I subclassed ClickableSpan because it’s rendered at the right time and because it nicely selects the entire span when you touch or click on the text. I overrode the onClick to do nothing because I actually don’t want to do anything with clicks here. Finally, I set the drawing context to underline the text.

Interestingly the framework is extremely limited by the choices available for rendering or drawing in the edit field. Most of the rendering is hardcoded or uses explicit choices (like setUnderlineText). While you can render images on the beginning of a line or completely behind the text, it’s harder to render graphics around or with padding and there is no span that allows the object to entirely draw itself unless it replaces the text (like an emoticon).

If you’re struggling with the span concept think of how it would be applied to rendering an HTML page. That helped me make sense of what drove the current design decisions. Take a wander through the span classes in android.text.style to get a better idea of what is supported. Also look at TextPaint and it’s inherited methods to see what can be modified in terms of rendering.

Scrolling or moving the cursor

It’s great to have the underlining and pretty names (and still get access to the underlying object), but we really wanted this to feel right. One behavior that was important to me was that if I move the trackball or use d-pad arrows to move the cursor through the edit field, it should highlight the entire Recipient, rather than move the cursor one letter at a time through the text.

While I could have captured the keydown/keyup and trackball events and responded to them, I found that the edit controls take a MovementMethod class that need only respond to certain directions. The documentation for the interface itself says it “should not be implemented directly by applications.” I took that to mean it was OK to subclass the ScrollingMovementMethod class that handles scrolling content within an edit field. I ended up coding something similar to how the LinkMovementMethod works, except that I made it work more like I expect.

The code is too lengthy to include here in it’s entirety but basically has two modes: left/right and up/down. In both cases I find all the RecipientSpans that are in the visible text region. For left or right, I find the span or word character immediately before or after the cursor, respectively, and select it. Moving left looks like this:

                int beststart, bestend;

                beststart = -1;
                bestend = -1;

                for (ClickableSpan candidate1 : candidates) {
                    int end = buffer.getSpanEnd(candidate1);

                    if (end < selEnd) {
                        if (end > bestend) {
                            beststart = buffer.getSpanStart(candidate1);
                            bestend = end;
                        }
                    }
                }

                if (beststart >= 0) {
                    if (selStart - bestend > 0 && WORDS.matcher(TextUtils.substring(buffer, bestend, selStart)).find()) {
                        Selection.setSelection(buffer, selStart - 1);
                    } else {
                        Selection.setSelection(buffer, bestend, beststart);
                    }
                    return true;
                }

The WORDS constant is a pre-compiled regex Pattern to look for any ‘word’ characters (Pattern.compile("\\w")). I used this so that when the cursor gets to the edge of a Recipients name, it skips the ‘, ‘ separator and moves to the next RecipientSpan or any email address typed in.

For up and down movements, I wanted to have the cursor move to the previous or next line and select the recipient whose name was above or below the current cursor location. This differs from the LinkMovement implementation where ‘up’ movements is the same as ‘left’ and ‘down’ is the same as ‘right’. In this case I parse the text in the target line and look for any spans to select. Here’s what the up movement looks like:

                int lineUp = Math.max(currentLine - 1, 0);
                int offUp = layout.getOffsetForHorizontal(lineUp, layout.getPrimaryHorizontal(selStart));

                ClickableSpan[] linkUp = buffer.getSpans(offUp, offUp, ClickableSpan.class);

                if (lineUp == currentLine && selStart > first) {
                    Selection.setSelection(buffer, selStart - 1);
                } else if (linkUp.length != 0) {
                    Selection.setSelection(buffer,
                            buffer.getSpanStart(linkUp[0]),
                            buffer.getSpanEnd(linkUp[0]));
                } else {
                    Selection.setSelection(buffer, offUp, offUp);
                }

Handling deletes

Having made movement through the field feel right, the next step was to handle deletes. If a RecipientSpan is selected and I press delete on the keyboard it will delete the recipient as expected. However, if I just have a blinky cursor and start deleting characters I want to have it delete the recipient as a single entity. I handle this by looking for keystrokes with a View.OnKeyListener.

                if (view instanceof EditText) {
                    Editable buffer = ((EditText) view).getText();
                    // If the cursor is at the end of a RecipientSpan then remove the whole span
                    int start = Selection.getSelectionStart(buffer);
                    int end = Selection.getSelectionEnd(buffer);
                    if (start == end) {
                        Recipient.RecipientSpan[] link = buffer.getSpans(start, end, Recipient.RecipientSpan.class);
                        if (link.length > 0) {
                            buffer.replace(
                                    buffer.getSpanStart(link[0]),
                                    buffer.getSpanEnd(link[0]),
                                    ""
                            );
                            buffer.removeSpan(link[0]);
                            return true;
                        }
                    }
                }

This is pretty straightforward — look for any RecipientSpans in the selection (i.e. where the cursor is), if one is found remove the visible text and the span. While I only look for a single span here (because I don’t expect to have overlapping spans) in another implementation it might makes sense to loop through and remove all spans that are returned.

Android: Hide a view when the soft keyboard is showing

This took far too long to find, mostly because the documentation doesn’t make sense (to me) and because the one hit on the android-developers list looked like the wrong answer, when it fact it’s right.

On the NPR News for Android app, we have a player at the bottom of the screen. This is handled with a FrameLayout in activity layouts. For activities that had an EditText field, the keyboard would push the player up and cover the important parts of the screen. See the image at right for the ‘before’ issue.

I can see this being a major problem for apps that host a banner ad at the bottom of pages (and saw a post on StackOverflow to that effect). Clearly you don’t want the ad to cover the content when your user is trying to type.

The solution was simple enough for us: mark the activity with windowSoftInputMethod = adjustPan in the Manifest and this resolved it.

    <activity
      android:name="SearchResultsActivity"
      android:windowSoftInputMode="adjustPan"/>

Unfortunately, windowSoftInputMethod is not very well documented. The reference says that there are two possible choices, adjustPan and adjustResize and that activities should always select one or the other.

adjustResize
The activity’s main window is always resized to make room for the soft keyboard on screen.
adjustPan
The activity’s main window is not resized to make room for the soft keyboard. Rather, the contents of the window are automatically panned so that the current focus is never obscured by the keyboard and users can always see what they are typing. This is generally less desirable than resizing, because the user may need to close the soft keyboard to get at and interact with obscured parts of the window.

What should be made more clear is that “resize” doesn’t mean scale or zoom here, it means that the container for your layout is compressed and every element is realigned to try to make it fit best. While this may make sense for applications that have content above or below the edit field that they want visible while editing, it isn’t what we were looking for.

By using adjustPan the layout manager leaves the activity as it was and only shows a portion of it — the part that would be visible if the keyboard were laid over the activity. Exactly what we want.

Snapshots and the New Android AVD Manager

So one of the biggest pains to mobile development with emulators is that starting up the emulator takes forever. Android tools version 9 addressed this finally by letting you take a snapshot of the emulator, just like you would do with a virtual machine. Here’s how I got my emulators really snappy.

First on the initial launch make sure you select Save to snapshot. Get the emulator up and running and put it into the state you want. I like to have it start at the desktop (home screen) view. Now close the emulator. I found that it wasn’t very responsive when closing, so wait a bit, watch the mouse icon go to “busy” and let it write everything to disk.

Now, launch the emulator again and this time uncheck Save to snapshot but make sure Launch from snapshot is chosen. Voila! The emulator launches quickly. And when you save it it shuts down quickly because it’s not saving anymore.

One caveat I’ve found with the emulator. When moving from network to network (i.e. one wifi point to another) for some reason the emulator loses Internet access (returning “host cannot be null” for HTTP requests). Unfortunately this means I have to have a save emulator snapshot for each of my common network and launch a new one, not from snapshot, when I’m on a new network. Anyone have any idea how to fix this?

Adding a custom jQuery-UI control to Formtastic in Ruby on Rails

A jQuery-UI slider rendered with FormtasticWhile working on OnCompare, over the last month, one of the controls we wanted to support was a slider, that let you choose between a range of values, as you can see in the example at right. jQuery-UI provides a slider control and Ruby on Rails already has jQuery built in (we’re using Rails 3). The challenge was that all of our controls are rendered from Formtastic.

Fortunately, Formtastic allows you to add new control types by extending Formtastic::SemanticFormBuilder. To do that I changed config/initializers/formtastic.rb to specify our own builder:

Formtastic::SemanticFormHelper.builder = Formtastic::OnCompareFormBuilder

Then I just needed to add a method that responds to :slider type.

class OnCompareFormBuilder < Formtastic::SemanticFormBuilder

  def slider_input(method, options = {})
    collection   = find_collection_for_column(method, options)
    html_options = strip_formtastic_options(options).merge(options.delete(:input_html) || {})

    input_id = generate_html_id(method,'')
    slider_id = "#{input_id}_slider"
    label_id = "#{input_id}_label"
    value_items = []
    label_items = []

    collection.each do |c|
      label_items << (c.is_a?(Array) ? c.first : c)
      value = c.is_a?(Array) ? c.last  : c
      value_items << value
    end

    label_options = options_for_label(options).merge(:input_name => input_id)
    label_options[:for] ||= html_options[:id]

    script_content = "$(function() {
        var option_values = [#{value_items.map {|v|"'#{v.to_s.gsub(/[']/, '\\\\\'')}'"}.join(',')}]
        var option_labels = [#{label_items.map {|l|"'#{l.to_s.gsub(/[']/, '\\\\\'')}'"}.join(',')}]
        $( '##{slider_id}' ).slider({
            value:100,
            min: 0,
            max: #{collection.count - 1},
            step: 1,
            slide: function( event, ui ) {
                $( '##{input_id}' ).val(
                        option_values[ui.value] );
                $( '##{label_id}' ).text(
                        option_labels[ui.value] );
            }
        });
        for(var i = 0; i < option_values.length; i++) {
          if(option_values[i] == $( '##{input_id}' ).val()) $( '##{slider_id}' ).slider( 'value', i )
        }
        $( '##{label_id}' ).text( option_labels[$( '##{slider_id}' ).slider( 'value' )] );
    });"

    label(method, label_options) <<
    template.content_tag(:script, Formtastic::Util.html_safe(script_content), :type => 'text/javascript') <<
    template.content_tag(:div,
        template.content_tag(:div, label_items[label_items.count - 1], :class => 'right-label') <<
        template.content_tag(:div, label_items[0], :class => 'left-label') <<
        template.content_tag(:div, nil, html_options.merge(:id => slider_id)), :class => 'slider-holder') <<
      hidden_input(method, options.delete(:as)
    ) <<
    template.content_tag(:span, nil, :id => label_id)
  end

end

Finally, I generate a ruby call in the ERB that adds a slider using something like this:

<%= f.input :value, :as => :slider, :label => question.text, :hint => question.description %>