Postmortem: 2 hours of downtime on October 23rd 2014

We encountered a downtime of roughly 2 hours today. It lasted from about 5:45am to 7:45am CET (8:45pm PST to 10:45pm PST, or 1:45pm to 3:45pm Sydney time).

We’re very sorry for the inconvenience it caused. Here’s what happened and what we’ll improve:

Cause and solution
The problem was that we had exceeded our daily budget for our servers, so the application started failing with a “quota exceeded” exception, resulting in our generic error screen. Increasing the budget was the quick fix to get the servers back up and running.

Background
We had changed our performance settings a month ago, and we didn’t realize that this increased our daily spend a fair bit. We usually set a daily budget that’s 5 times higher than the average actual spend. But the performance settings increased our spend while we didn’t increase the maximum budget, so the ration dropped from 1:5 to 1:2. But everything worked fine until today, when all of a sudden our servers misbehaved: After a sudden increase in load the servers didn’t spin down as usual, but kept running idly for hours. Idle or not, they accumulated costs at a very high rate over several hours, until we finally exceeded the limit.

The error message was very generic, and in the first hour the site was still intermittently available, so it looked like a temporary glitch at first. Only when our US staff realized the problem didn’t go away, they started calling the Berlin dev team, which however was asleep, delaying things a bit.

The worst part of the problem occurred when the dev team realized that only a single person (me) had access to the billing settings to increase the budget, and my phone was on mute. It took some 30 minutes until the vibrate of my coworkers actually woke me. This is clearly unacceptable and the most embarrasing part of the story.

Fixes and longer term improvements

As you can see, quite a few things went wrong, and the problem could have been caught or at least mitigated earlier. Things we’ve done or are doing to prevent a similar case:

  • We’re in touch with a Google representative to figure out why our servers were costing us so much more money despite idling. This is the main reason for the downtime and we need to adress it to reduce our bills.
  • We’ve reverted the performance change and doubled our budget, so our safety factor is now 1:10 instead of 1:5
  • Additional staff can now adjust the daily server budget to provide a quick fix in case a similar problem strikes. We’ve been quite good at removing single points of failure so far, but we entirely missed this one.
  • We’ve added land-line numbers to our internal contacts list, since mobile phones are just too prone to be muted, have dead batteries, or just be in the wrong room.
  • We’ll adding automatic early warnings for SI staff if the spend is approaching 50% of the daily budget the moment our platform supports these kinds of queries. Unfortunately it’s not yet possible to automate this due to Google App Engine limitations. Until it becomes available, we’ll look at our average spend a lot more closely.

Affected client reimbursement
We don’t provide formal SLAs but we take these kinds of downtime really serious and will reimburse affected clients.

If you’re a client and you’ve been affected during an important phase of your project, please let us know at support@small-improvements.com, and we’ll either reimburse you for this month, or extend your license by a free month, whichever you prefer.

Let’s meet up! USA tour 2014

It’s the high season for HR conferences! We’ll be attending the HR Tech Conference in Las Vegas (October 7th-9th) as well as HR Evolution in Dallas (November 8th), which Small Improvements is sponsoring. If you are also there, please say hi!

Like last year, while on the road, it makes sense to get as many Small Improvers together as we can for a fun time! We’re hosting a few Meet&Mingles across the U.S. to bring customers together, exchange HR ideas as well as reconnect or meet some of you for the first time.

  • DMeet+mingle_Denver_2014-01enver/Boulder, Wednesday October 15th, 6pm onwards: SendGrid has been very generous in offering to host our event at their Boulder office. This should be an awesome event since most of our team will be in town and attend also!
  • New York, Tuesday October 21st, 6pm onwards: We’ve chosen the Loreley Biergarten for specializing in German beer and sausages. Small reference to our roots. Needless to say, should fun!
  • San Francisco, Monday October 27th, 6pm onwards: Save the date! The venue has yet to be determined, but we will update the blog when we arrive at one.

If you happen to be in any of those areas and are keen to stop by for a drink, please get in touch! We’d be thrilled to put you on the guest list and meet you.

Meet & Mingle event in London

A Meet & Mingle is Small Improvements’ client event. What started as a one off on a trip to San Francisco is now a regular event that we’ve held in 6 cities across the US and Australia.

The concept is simple: We invite customers and prospects to meet up in person, have a drink on us and mingle. The feedback we are getting is awesome since it is a casual after work get together – meet & great, happy hour or whatever you want to call it. And for us it is a lot of fun to put a face to a name and introduce like minded HR professionals to each other.

On Wednesday August 27th, we’re holding our first European Meet & Mingle in a cool bar in London!

Meet+mingle_London_2014

Read a little more about Meet & Mingles and check out some pics from our last event in Sydney. And should you like to join us, please reach out to Linda.

Much looking forward to catching up with outstanding HR leaders in London soon!

The Small Improvements miracle story – WIRED magazine

Wired_articleWe love a good mention! But WIRED took it a little too far.

According to an article, ‘Small Improvements reported €280 million in revenue last year’. An astonishingly high number for a bootstrapped startup with 230 clients, that instantly sparked interest from investors. It would mean that every client pays well over €1 million a year, which would be hard to achieve with our lean pricing model and limit of 2000 users per customer… Certainly this report does not come from us!

We are actually quite transparent with our revenue and blog about our start-up numbers. To bring the stats of this article up to date:

  • Over 320 clients on 6 continents
  • A team of 13
  • Located in New York, San Francisco, Sydney and Berlin
  • Around $1 million in revenue last year

That is not quite the miracle story from WIRED but still a great success for our small venture! Should you have further questions about Small Improvements, please do reach out. As mentioned we are happy about media exposure but prefer a more accurate reflection ;)

Join us for our Sommerfest in Berlin!

488047_512711482104740_2076152363_nSave the date: On July 17th 2014 from 7pm onwards, Small Improvements is hosting a party and you are invited!

 

Join us in our HQ in Berlin (Exerzierstrasse 24, 13357 Berlin) and meet each and every member of our team! That’s right, we are all flying in and it will actually be the first time for all of us to meet in person too! With Scott and James from San Francisco, Ivo from New York, Chris and Linda from Sydney and even our lovely advisor Anton from London, our office might be bursting at the seams but thankfully we have a courtyard to our disposal also.

 

Please join us in celebrating summer with friends. Drinks and food are a given (did I get that order right?) and geeky conversations on all things tech and HR unavoidable ;)

 

For catering purposes, please let us know if you can make it and how many friends you are bringing. Much looking forward to opening our doors and having a fun evening with you.

 

Mobile app in the making

Every software service today needs a mobile app, right? But is it good enough to have an app just for the sake of having one?

The Background

We actually started working on a mobile version three years ago but decided to stop and focus solely on our core product first. Now we are ready to tackle mobile properly! We’ve written a new Small Improvements RESTful API which makes the transition to mobile easier, smoother, and sustainable for rapid co-development. iOS will be our first platform.

The Beta Version

The use-case we are focusing on for the beta version is: Capturing and referring to ongoing feedback on the go. Say a manager goes into a meeting with his direct report and wants to see what was last discussed, or someone attending an outstanding presentation quickly wants to post public praise or take a private note. To support this, the beta version will focus on the messages feature as well as a offer full overview of colleagues and direct reports. Performance reviews, objectives and todos are displayed for reference.

Mobile_app_visual_Messages Mobile_app_visual_taking_notes Mobile_app_visual_self_assessment

The functionality is a streamlined version of the web application so anyone who has used SI will be familiar with the set up. A new addition to the four feature tabs is the ‘people’ tab. In here, you can search for colleagues, access your direct reports’ profile pages quickly, as well as check your todos.

Mobile_app_visual_people Mobile_app_visual_Profile

The message feature is pretty much fully functioning with default posting categories, voting, and replying to messages. The reviews are currently limited to viewing only. We do not plan to support any of the administrative functions of SI in the mobile app.

Let us know your thoughts! There is no ETA yet, but if you have an iPhone and would like to take part in the beta testing, please get in touch. We want to make sure we deliver the right app that supports your most common needs.

Introducing 2-step verification in Small Improvements

We’re proud to announce our newest security feature: You can now connect your Small Improvements account to your mobile phone, and then when you log in with a new browser for the first time, you’ll be asked to produce the SMS token (or app-generated token) to prove that you actually have the phone. You only do this once per browser of course.

This little step increases security a lot: Imagine a hacker were able to obtain your password elsewhere (e.g. from a hacked online-service you used). Without 2-step verification, they can log into all of your accounts easily. But they can not log in to SI (nor other 2-step protected services) because they will have to enter the SMS token as well – but they don’t have your phone, so they can’t receive/generate the token.

Now, you don’t have to force all your SI end users to enable 2-step verification. But it would be a good idea to enable it for all your admin staff, and maybe for the CEO and CFO, because those users have access to a lot of confidential data and would make great targets for hackers.

You’ll find more about this feature in our documentation page.

Amazing new admin screens

We decided to entirely rewrite some of our administration screens. We started by creating new versions of the performance review and 360 review overview screens, and the results are impressive! We’ve been beta-testing the features with a few clients, and new evaluators since March have had access to the screens as well, so we’re now proud to announce their public availability.

The “classic” screens are still available, you’ll find a link in the top right corner. If you feel you’d like the old screens back until your current review cycle is over, please send us a mail. (We don’t really see any reason why you’d want the old screens back, but it’s possible)

Video

A lot of the improvements are about speed and usability, so the video explains it best!

Screenshots

For those of you who are in a hurry, here are some screenshots!

Cleaner layout

We’ve moved all admin actions to the top left.

overview

Improved cycle picker:

The cycle picker is now part of the header, groups all cycles and provides additional detail.

cycle-picker

Admin actions to the top right:

The admin actions have been grouped by topic, and we’ve added a few new ways to notify your staff.
admin-actions

Email preview before sending bulk mail:

Before sending a mail to 500 employees, it’s advisable to double check your mail content. Now that’s part of the normal workflow.
email-preview

All new “add reviewees” screen

This is the best part of the new screens: A org chart view of your company helps you decide who should get reviewed within the current cycle. You can also switch to list view without losing your selection, and sort by name or hiring date:
add-reviewees

We’ve also improved tons of smaller details, like the ability to sort by each column, a new distribution of ratings, and more accessible timeline view. But you’ll have to experience it! :) In case you run into any unexpected problems with the new screens, the old screens are still available, you can return to them by clicking the “return to old screen” link in the top right corner.

 

Roadmap

We have also started working on a new version of the user directory, and we’ll continue to iterate on the overview screens because we still have lots of plans. Let us know your thoughts!

 

 

Heartbleed security update

Short version:

A security error (nicknamed Heartbleed) in the SSL protocol has affected a large share of internet services and devices. Small Improvements runs on Google data centers that were vulnerable to the Heartbleed error. All SI systems have been patched by Google by Wednesday, and we have reissued all SSL certificates on Thursday. We’re not aware of any actual compromise or breach, but as a security measure we have reset all user passwords on Friday.

 

Background

Small Improvements is hosted on a service called “Google App Engine”, and unfortunately Google App Engine was affected by the co-called Heartbleed bug. On Wednesday Google issued a statement that all systems had been secured. We had taken a look on Tuesday when we first learned about the problem, and didn’t find the SI infrastructure to be vulnerable then already.

 

Assessment

It was good news that our service was patched early on. But we still had to wait until Google officially announced that we’re safe before we could replace our SSL certificates, so that’s why it took until Thursday.

Having been vulnerable doesn’t necessarily mean that anyone actually stole any data from the SI systems. But due to the nature of Heartbleed it’s impossible to know. This applies to all the thousands of services affected by Heartbleed by the way. We believe it’s unlikely that someone singled out Small Improvements specifically, but it’s entirely possible that some attackers started harvesting data from the internet in general, and that these people are now sifting through whatever they could grab.

A key concern in data breaches are passwords and confidential data in general. We always store passwords in an encrypted form (using bcrypt) and all textual content it stored using AES-256, so it’s unlikely that anyone would be able to decode this data even if extracted from our servers. But due to the widespread nature of Heartbleed, and the fact that many people use the same passwords across websites, we feel that there is a significant risk of passwords stolen elsewhere being tried out on Small Improvements accounts some time in the future. SI admins have access to a lot of confidential data, but even non-admins store quite sensitive data in SI, and we don’t want any of it to get exposed.

We first intended to prompt our users to change passwords on their own. But people have other work to do, might be on vacation or simply not read our mail. So we then decided to clear all SI passwords at once on behalf of users. Passwords can be reset using the “forgot password” feature via an email. So while this bulk reset is slightly inconvenient for end users, it removes chances of a future hacking attempt.

Next steps for SI admins

We have taken security matters very seriously at SI right form the start of our company. We’re continuously adding internal security improvements, and have also shipped a couple of customer-facing security improvements recently. Our latest development is a 2-step verification feature (also known as 2-factor auth, or mobile token auth). This has been in private beta for a month, and it’s now entering public beta. We’d like to encourage you to try out the feature (and enable it at least for administrators). Also, you can use IP range restrictions to limit access to SI to your company VPN. And you should consider increasing the minimum password length for users so far that they are forced to use a password manager (and thus have no reason to reuse passwords across sites at all anymore).

In other news, we’ve also increased some internal security triggers to be a bit more “trigger-happy”. For instance, even a minor update of your browser (as per your user agent) will now require a user to re-login. We have plenty of plans for additional internal and external improvements, and we’re very open for your suggestions too!

Next steps for everyone

Although many vendors try to use very evasive language to play down the vulnerablity (“all patched, all is good”), we believe this bug should be taken very seriously by everyone. Many of your passwords, both for business and for private use, may have been compromised, and get used against you either now or some time in the future. You should reset all your passwords, starting with your most important services. We recommend keeping your very most important password in your head, and for all the non-critical passwords download a password manager today. It will allow you to use a new password on each service you use, limiting any future hacking impact as well.

 

More information

Details at CERT: http://www.kb.cert.org/vuls/id/720951

The Heartbleed bug website: http://heartbleed.com/

A statement from OpenSSL: https://www.openssl.org/news/secadv_20140407.txt

Heartbleed Bug explained as a cartoon:  http://xkcd.com/1354/

Meet & Mingle event in Sydney

It’s been a year since our last Meet & Mingle in Sydney so it is time to reconnect over drinks! Join us for evening of networking, exchanging ideas and meeting Small Improvements’ newest Sydney-sider Chris. Drinks on us!

The event will be on Wednesday April 2nd.

If you are an existing client and have not replied to my email yet, please get in touch to not miss the opportunity of meeting up with fellow awesome HR leaders in your hood. Should you be new to Small Improvements and interested plus happen to be in town, please reach out to Linda.

Looking forward to a great evening with you!

Image

Performance Pain Points – The Prescription

perf_pain_presecription

Previously, I played the part of PM-D…Performance Management Doctor (The Diagnosis). I talked about the symptoms of an unhealthy performance management process based on feedback from the Impact99 HR Summit Toronto. I had a closer look at the pain points that people described and realized that they, and the remedies, followed a path from start to finish.

Traditionally, most organizations fail to communicate the true purpose and expectations of the performance review process. Therefore, most employees consider it to be a negative experience; waiting to hear about how they have not achieved the expected level of performance. By default, most people see the process as merely a means to an end…that end being salary increases or promotions. What do you consider to be the purpose of the process? What are your expectations? Have you clearly communicated those expectations to your entire organization? If your people expect one thing but receive another, there is a disconnect that undermines the process.

So, let’s ask ourselves what connects people to the process? Is your process one that is typically pushed from the top down?  Even if it isn’t, is there a perception that it is?  For example, what  would your employees say if you asked them “who owns the performance process, who benefits from it and how?”  Would they see themselves in the process?  Would they see themselves as the owners, the drivers, the ultimate beneficiaries? If everyone takes ownership of the process – exec’s, HR, managers & employees – then it will cease to be a process that people “have to do” and will become something that people are motivated to do because it’s their own! It becomes part of the culture.

Transparency, reciprocal trust, collaboration and alignment result from ongoing discussions and will also draw people into the process. Discussions about the organizational vision results in productivity and pride in one’s work.  Communicating expectations assists in setting reasonable objectives. Someone who is enrolled in the business, who has clear objectives, and understands their contribution is someone who is eager to set their goals and measure their results in a performance review.  If the performance process relies on open discussions so that nobody is left in the dark, and there are no surprises at review meetings.

As I mentioned in Feedback: The Guidance System for Performancemore frequent performance check-ins are a much better use of time and effort than the traditional “year-end review” meeting. People perform on a daily basis, so why wait until the end of the year to discuss accomplishments and challenges. There’s no better time for improvement than the present. Organizations and individuals can become more agile by making small adjustments in their performance as they go along…not 10 months from now!

OK…you’ve just had your performance check-in with your manager. You’ve discussed your accomplishments, areas for improvement and set your objectives going forward. Now what? You have to keep them connected to the process; you need to follow-up. This shows integrity – aligning one’s actions with their words – and reinforces the trust you created through ongoing dialogue. If a need for more training was requested by the employee or manager, it would be a colossal fail if nothing was provided. If an employee has set an objective and doesn’t formalize it and provide updates…#fail again.

Expectations, ownership, discussions, check-ins and follow-up. Combine all these things and you have a process that is productive and meaningful. People within an organization want to know that their performance process actually means something, individually and for the bigger picture. Especially the younger generation. Organizations need to realize that this generation will expect meaningful work, which includes ongoing feedback and discussions. They will want to know how they are performing and how they can continue to perform at their highest level.

Wrap all this up in a simple process where individuals and teams are aligned with organizational goals, and people will be confident that they are part of something bigger. Best of all, you can elevate the level of overall engagement. A great performance process shows that the organization’s leaders actually care about their people. Recent studies (like the 2013 Spring report from Globoforce) have shown that leaders recognize engagement as one of the top challenges within their organization.

We are constantly looking for ways to improve the way we manage our performance. We look for new and innovative tools and methods to increase engagement and productivity. Sometimes the best thing for our organization is right under our noses. Try writing your own prescription.

Meet & Mingle Melbourne

While the Australian Open is on and Melbourne is hit by a heat wave, there is no better way for HR folks to relax then to escape into a cool environment and meet & mingle! Don’t get me wrong, I love Tennis and am looking forward to being in the Rod Laver arena myself next week. But on Tuesday January 21st from 5pm onwards, Small Improvements will serve drinks! Advantage you :)

Come and join our famous Meet & Mingle event, where clients and prospects get to geek out about performance management and simply hang out together! It’s a casual event without an agenda. So if you are free and keen, drop Linda a line for more details.

Meet+mingle_Melbourne_2014-01

Performance Pain Points – The diagnosis

11588728936_b627b5ec7a_cAlthough not recognized by any medical association, I’ll play the part of the PM-Dr… Performance Management Doctor.

A sampling of “patients” at Impact99 HR Summit 2013 Toronto described several symptoms of an ailing performance management process. There appears to be a deficiency of several important elements for a healthy process, such as results, expectations, meaning and consistency.

pm1

pm3

Read the full post »

We’re phasing out support for Internet Explorer 8

We’ll be phasing out support for Internet Explorer 8 over the next couple of months, so here’s what you need to know:

IE8 got released in March 2009. By Internet standards that’s very old. It was the first decent browser by Microsoft in a long time, but even back then it was not a great browser. And compared to today’s versions of Chrome, Firefox and even IE 10 it’s slow, and it’s a pain to develop software for it. This frequently slows us down at Small Improvements. The tools we use to build SI are dropping IE8 support, making it even harder to continue supporting it.

So starting in January, we’ll phase out support for IE8 gradually, and we’ll make IE9 the minimum Internet Explorer for Small Improvements. This won’t happen in a big bang. We’re aware that many of our customers are in the middle of their review cycles, and we won’t slam the door shut on them just like that. Instead, we’re starting to ship new admin features that require IE9 now, and we’ll keep alternatives open for a few more months. Once we drop these alternatives, administrators and middle managers will have to use newer browsers, but end users can still work on IE8 for a limited time.

For example, we have been re-implementing several administration/middle-managers screens using new technology, and these new screens will not work in IE8 anymore. We’ll ship three of these new overview screens in January. We’ll keep the current (“old”) admin screens alive for another few months, so you can opt to not use the new improved screens for now. But this option will go away around April 2014.

End user screens are next. We’ll re-implement the messaging and the objectives screens in spring, and we’ll roll these out around June 2014. That’s the time when even your end users will need to have a more modern browser to access SI for messaging and objective-setting. We expect to have reimplemented most remaining end user features (reviews and 360s) by Septemer, and that’s when IE8 simply stops working entirely with SI.

So in summary, dropping IE8 won’t happen over night, but in case you still use IE8, now is the time to consider how to get your admins, then your managers, and then your end users to using modern and secure browsers. It’s not that hard really. Even if you’re stuck on Windows XP for now and cannot install IE9, there’s always Firefox and Chrome that work just fine on XP.

New – Video Testimonials

Editing testimonial videosCustomer testimonials offer a great insight into what actual user think about a product. We are very proud of our customer success stories that serve as little case studies. And now we are are adding videos!

As part of a 7 week trip around North America, we visited lots of customers and asked how they are going with Small Improvements. Not only did we get invaluable feedback, but while at it, we also pointed a camera or two at them. We’re so pleased with our ‘home made’ videos, that we are now filming customers in Australia. Maybe Europe is next and I’m certainly not opposed to the idea of flying to all 6 continents that we have clients on ;)

Check out the first video below plus follow our testimonial overview page for plenty more to come! We’re currently busy editing so are looking forward to your thoughts!

Follow

Get every new post delivered to your Inbox.

Join 27 other followers