Postmortem: intermittent mail delivery on July 20th and 21st, 2015

On Monday and Tuesday this week, we experienced intermittent mail failures that resulted in many notifications from Small Improvements not being delivered. The failures started around 1:00pm CEST and continued till around 5:00pm CEST the next day (July 20th, 4:00am PDT – July 21st, 8:00am PDT).

We are sorry for any inconvenience that may have been caused, and we have taken steps to mitigate similar problems in the future.

Cause and solution

The cause of the outage was our application servers hitting a hidden limit that only affected outgoing network traffic like email. This meant that the application behaved normally for customers visiting the website, but it also set up a perfect storm. Because our outgoing network traffic was not working, we did not receive any of the notifications that were trying to report the failure.

We have now adjusted the application so that email sent from Small Improvements does not get affected by this particular limit.

Background

We are long-time users of the Google AppEngine platform, which allows us to easily scale our application to serve large amounts of data to customers all over the world. The trade-off of this massive automated scalability is that we have to run our application within a proprietary sandbox. The problem occurred due to our application servers exceeding a daily quota applied to our sandbox. The specific limit that we hit was an undocumented, low-level network operation that is not displayed in any of the standard monitoring tools Google provides to its customers.

To make matters worse, this particular limit did not just block email to customers; it also blocked our internal alerting system that ordinarily notifies our development team of any problems. The icing on the cake was that our external logging system was no longer being updated due to the limit, so we did not see any errors there either. This series of unfortunate events resulted in the worst-case scenario where our customers noticed a problem before we did.

Fixes and longer term improvements

As soon as we realized what had happened, we contacted Google and had the limit temporarily lifted, which restored email delivery. Meanwhile our development team went to work on modifying our application to send email using a different mechanism that would avoid triggering the limit in the future.

  • We are continuing our contact with Google to find out why this limit only just recently began to be triggered.
  • We have already deployed a change to our email subsystem to avoid usage of the quota-limited functionality.
  • We have begun an audit of our application integrations to ensure that other features could not be affected by a similar limit.
  • We have tuned the alerts delivered by our fallback error reporting service so that a failure in the email subsystem will still be reported to our developers.

We have already contacted the companies who were most seriously affected by the outage. Once more, we sincerely apologize to everyone affected.

Proudly announcing the launch of Small Improvements iOS mobile app!

Want to praise a colleague while in a meeting, take a note on the train or double check progress of objectives on the way to a 1:1? Then you’ll love our new app! It’s been build especially with managers in mind to help capture ongoing feedback as well as access details on the go. Just go to the app store and download it.

Overview

Mobile_app_visual_profile_objThe mobile app follows the flow of the web application with tabs for messages, reviews, 360 degree feedback as well as objectives. The ‘people tab’ is a new addition to the mobile app. In ‘people’, you can search for colleagues, see your own profile and to do’s as well as access all details of your direct reports easily. Profile pages allow for a comprehensive overview and you can take notes about your colleagues or communicate right from there too.

Messages like praise, feedback and meeting notes are important ingredients of ongoing conversations. With the mobile app you now have a great overview, can like, reply and write new messages on the go. The 360 degree feedback, performance reviews and objectives tabs are for your reference only for now. You can view your latest feedback and check the status of your objectives but not write your review or set up new goals a the moment. We feel that these actions are still rather done on a computer than on the phone.

 Mobile_app_visual_messages   Mobile_app_visual_360   Mobile_app_visual_reviews

Check out more details regarding the app, try it out and get in touch! We’re much looking forward to your feedback and thoughts. Happy ‘small-improving’ on the go :)

New user directory tab, plus a “weighted objectives” helper

We’re proud to announce two minor improvements today:

New Gridview tab in the user directory

We’ve added a new tab to the user directory that allows for easier filtering by categories such as department, location and role. At the same time, this tab puts a lot more emphasis on the user’s profile picture, helping you figure out “who was this person from marketing I recently met in the hallway?”. The new tab is placed to the right of the existing ones:

User_directory_-_Small_Improvements

 

New objectives weightings popup

Another nice improvement was added in the objectives overview. If you’re using the weighting feature, then all objectives within a cycle should add up to 100%. If not, we’ve always displayed a little warning hint. As of today, we also provide a very interactive helper screen to ensure your combined weightings equal 100%:

Objective_Overview_of__Objectives_Q2_2015__-_Small_Improvements

You can trigger that screen from the warning hint underneath each person’s objectives.

But, is this newsworthy?

Admittedly, the two screens are not game changers. The interesting point is, these two features were developed by new staff that joined Small Improvements a mere 3 weeks ago. Similar improvements were made to our backoffice toolset, and many more improvements are being worked on: Expect a neat “mini user profile” popup, user-statistics displayed over time, a lightning fast edit-user overlay, and so on in the next couple of weeks.

The best thing is, our work on the main features isn’t impacted. We’re currently wrapping up the objectives feature overhaul, and we’re hoping to roll it out to evaluators in two weeks, and then to existing customers a few weeks later. The cycle admin screens are up for a refresh next, and we’re also rewriting our user-importers, making CSV/Excel userimports available as self-service feature soon. There’s a huge list of large tasks we’ll work on, but the nice thing is that our ability to slot in additional small improvements just increased dramatically.

We improved the messaging feature: Mentions, filtering, design and more

Messaging was the first feature we implemented into SI. The feature itself stood the test of time well, and we’re quite proud that modern companies such as Buffer are using a feature that we designed and implemented over 4 years ago. But the user interface was starting to look dated, so we’ve just rolled out several improvements to the stream:

We’ve streamlined the User Interface

We’ve moved the author and recipient to the top of the message,t he upvotes from the far right underneath the message, and the action-buttons are now always visible (no need to hover). The message width was slightly reduced, while the font size was slightly increased, meaning that messages are more readable now. Here’s what it looks like now:

Messages_-_Small_Improvements

You can now @mention users!

This improvement is very neat: By default only very few people get notified immediately by mail when a message is created: The recipient of a message, and the people who a message is restricted to (if any). But by mentioning a person (using the ‘@name’ format you can trigger a notification to that person as well, which is a very handy feature:

Messages_-_Small_Improvements

We’ve added a simple filter option on the right

Thanks to the slimmer message display, we can now use the right hand side for a new filter option. You can now limit display to certain kinds of messages (only praise, or only announcements for instance), or you can display only messages by or about a person.  You can combine these filters, showing only messages by person A about person B in a certain context.

Messages_-_Small_Improvements

We’ve dropped downvotes

Downvotes were rarely used by our clients. Downvotes can be great to quickly identify illegal or inappropriate content on platforms like YouTube or Reddit, but ultimately SI is a closed environment and the need for downvotes is minimal. So we removed the button from the UI. Existing downvotes are still being displayed, but no new ones can be added.

There’s now only one upvote per person, and it’s rebranded into “like”.

Just like Facebook, enough said. Or well, let’s elaborate just a bit: The original intent was to allow multi-upvotes to signal very stong agreement. However, it’s just very uncommon and many people got confused. It’s really not worth the confusion of many people to make a small minority happier, so one like per person it is now.

We’ve implemented endless scrolling

It’s hard to brag about this, because having to click “load more messages” was annoying in 2011 already. “Fixed at last” is probably the best way to describe this improvement :)

What’s next?

This overhaul was long overdue, and it improves the feature a lot. Still, some common feature suggestions are not covered yet. Favorite suggestions like “ability to praise multiple people”, “improved email notifications”, “better display on user profiles and on the dashboard” are still on our roadmap. So this was just the first steps in many.

We’re currently busy overhauling the objectives feature, and the improved messaging UI will make commenting objectives a breeze. We’ll work on vastly improved user profiles and dashboards soon, the messages will be tightly integrated on those of course. We are considering to allow commenting on reviews, so multiple stakeholders in addition to the actual reviewer can discuss a review in writing. We will definitely implement “multi-praises” and better emails at some point, but it’s too early to talk about an ETA for either of these yet.

Let us know what you think!

Now that the new messages UI is live, what do you think? Do you have feedback or suggestions for improvement? We’d love to hear! Simply send us a mail to support@small-improvements.com, we’ll route this to the designers and developers of the improved stream.

We rewrote our BambooHR Integration. Unfortunately it sent misleading digest mails instantly. Please disregard.

We’ve overhauled our BambooHR Integration, making an already successful integration even better.  Details below. Unfortunately, the code triggered confusing status mails to all admins at companies that were already using the BambooHR integration.

Please ignore the mails we just sent you

We took the new importer code live yesterday, and it was supposed to be a silent migration. Unfortunately, the new code triggered digest emails to many administrators, letting them know that SI had supposedly updated all their users. We’re now also importing the employee ID field, which is a good thing, but the digest didn’t communicate this well, and it should not have appeared in the initial import at all.  No “real” changes (like core fields or reporting structure) were made to your user base, so you can disregard the mail.

To make things worse, the digest accidentally listed a “deleted user” as the author of the change, not the “BambooHR Import” as it used to. We’re sorry about the confusion this has caused. In effect nothing happened and all is good.

Here’s why the new import is a great thing anyway

The BambooHR integration is our most important integration by far. It has been working reliably since 2011 and saves our customers tons of headaches by keeping BambooHR and SI in sync. We rewrote the integration to make a great feature even better. It is that central to us. Here’s what changed:

More interactive

Especially if you’re a larger company, you’ll love the fact that the preview table can be sorted by any column, and you can use the bar chart at the top to drill into the changes that you’re most interested in.bamboo2

Audit trail, and more secure

Each successful import now leaves an audit trail entry, so you can go back in time and get a detailed explanation what changed at what point. Click the details button for the nitty gritty of “what user changed in what way exactly?”

bamboo4

We’re storing your API key as before, but we’re not displaying it beyond the first couple of characters anymore. Not unlike a credit card you entered into an online shop, you’ll be able to identify which API key you’re using, while it’s not possible to extract it anymore, since only a few characters are displaying.Manage_the_BambooHR_Integration_-_Small_Improvements

More scalable

The old import had some inherent technical limitations that meant it could only preview up to 300 changes at once, which was especially annoying when a larger client wanted to preview the initial import. Now fixed. Preview as many changed users as you like.

More intuitive for new users

When visiting the screen for the first time, you only need to enter your credentials, and then you save them. Only then you’ll get to see the preview screen and the history tab, and you won’t end up in the config tab anymore by default, but in the history tab.

That’s how it should have been right from the start, but the “old” BambooHR Import had grown over time, and while it worked it was just a bit clunky to use. Now fixed.

Built with future integrations in mind

Truth be told, the old import was a technical mess internally. It worked fine, but we couldn’t reuse any of it for what we’re planning. So we made sure that new import is the perfect basis for future integration. We’ll rewrite the Google Apps integration next, and we’re aiming to allow self-service CSV and Excel import later this year. And we’re hoping to provide a convenient Workday import as well soonish.

As always we don’t promise any exact dates (our roadmaps do change) but we’ve come a long way already, so fingers crossed!

We’re streamlining our pricing: Less plans, better volume discounting

Short version: Starting April 15, we’re simplifying our pricing for new customers. All features are available under one plan at our previous Gold plan price of $6 per user/month. We’ve also expanded our volume discounts to 3 tiers at 200, 500, and 1000 users so Small Improvements remains affordable for our larger clients. Existing customers are unaffected, your price will never change, but it could make sense to upgrade for some.

One unified plan instead of three

Since our inception in 2011 we’ve had 3 payment plans: Bronze, Silver and Gold. Bronze had all the core features, Silver added many product integrations, and Gold featured convenience features such as custom design and single-sign-on. Our reasoning was that we wanted to give everyone a chance to use SI at a very low cost, and only charge for features that make SI even more awesome.

We’re now scrapping that model and dropping the Bronze and Silver plans. In a nutshell, we feel every user should experience the Gold plan goodness, or not use SI at all. Our tool has grown so comprehensive that $6 per user is even more of a bargain than 2 years ago, and that there is no point selling an artificially reduced plan.

We had expected that most Silver and Gold plan features would be attractive mainly from an administrative perspective (like SSO via Google Apps, or custom design). But in reality all Silver/Gold features are also great for end users (like SSO or custom design), and we don’t want end users to miss out. We’re growing by word of mouth, and our buyers are the most happy when their staff are happy. No point making staff less happy just so finance can save a few bucks. We want every user to love SI, be it the intern or the CEO, and artificially removing features is simply not a good idea.

New volume discount tiers

While at it, we’re also adjusting our volume discounts. Our “old pricing” had two tiers: Up until 500 users you paid $60 per user per year, and for every user above 500 you paid $30. We’re changing this to include 4 tiers.

Users Price per user per year Price per user per month (when paying monthly)
10-200 $60 $6
200-500 $40 $4
500-1000 $30 $3
1000-2000 $20 $2

It looks slightly more complicated, but even in the worst case (if you have over 1000 users) it’s just a sum of 4 values. For instance a company with 1500 staff would pay: 200*$6 + 300*$4 + 500*3 + 1000*$2, in total that’s $4900 USD per month. You can calculate your exact price on the pricing page too: http://www.small-improvements.com/pricing

Grandfathering: No change to existing customers

All existing customers can stay on their current plan for as long as they please. So if you signed up for the Bronze plan, you can stay with that until you quit using SI, whenever that is. You can switch to “old Silver” as well. We’re keeping the old pricing page online for your reference. You’re welcome to upgrade to the unified pricing whenever you want, though.

Only if you leave SI for more than 2 months and then come back for a renewal, the new pricing applies.

Transitioning period

If you signed up for a trial on or after April 15 then the new pricing applies instantly.

If you are testing SI already and have signed up for a trial account on April 14 or earlier, or you requested a quote already, then you can still get the “old pricing” until the end of May. From June 1 onwards, the new pricing applies to you as well.

Edge-cases and fine print

There will be no “refunds”. Meaning, if you signed up for 1000 users on the Gold plan in December 2014 for a year, you would have paid $45k. On the new pricing, this will only be $39k. There is no “refund” of $6k since you didn’t “overpay” – you simply paid a price that made sense at that time. This is similar to when you buy a new car and a month later it is offered at a lower price, your car dealer isn’t going to issue a refund. However, we don’t want you to feel unhappy either, so we’ll consider extending your license by a month or two for free if you’re very unhappy about our product getting better and cheaper at the same time. ;) Also, keep in mind that you will benefit from the new pricing when renewals are due.

If you are on Silver and have more than 400 users, then we’ll switch you to the unified plan automatically, since it’s around the same price point, and the unified plan has more features.

Please welcome the new Objectives overview screen

We’re proud to present an overhauled objectives overview screen: It follows the new visual style of the reviews and 360s overview, and is much more interactive than the “old” version. The new screen allows for easy sorting on each column, there’s the quick user search in the top right corner, and all admin actions have moved to the top right.

What’s more, we’ve added bar charts to display the distribution of objective categories, ratings and status (if you use them) This saves a lot of vertical screen estate compared to our old pie chart approach, and makes filtering a lot easier and accessible. The following cycle uses the status feature (but not the categories) so only one bar chart is showing. You can click any segment to filter.:

Objective_Overview_of__Objectives_Q1_2015__-_Small_Improvements

Continue reading