We’ll cut to the chase: Balanced, like almost everyone on the web, was affected by Heartbleed. No customer data was directly leaked by this vulnerability, and we’ve since patched our servers and rotated our keys.
If you use an official client library, we will be releasing some nice-to-have updates in the near future. If you’ve logged into Balanced from a public or unsecure wifi connection, you may want to rotate your API keys to be extra safe.
Now that that’s out of the way, let’s talk details.
You can read http://heartbleed.com/ for a full summary, but this paragraph explains it most succinctly:
The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users.
Unfortunately, the versions of OpenSSL that were affected have been used widely over the last two years, and so quite a large number of hosts were affected. :(
How it affected Balanced
If you read my previous post on Balanced’s
know that we are built using Service Oriented Architecture. This means that the primary hosts of
ours that are exposed to the broader internet are
https://api.balancedpayments.com and https://js.balancedpayments.com . Those
both go through
Midlr, so it’s really one set of servers that was affected.
Remember, Heartbleed allows you to read the memory of the affected server –
Midlr. The ELB essentially terminates SSL and forwards the request on,
the only sensitive information that is in RAM on
Midlr is the private key
for our SSL certificates.
So what happens when someone gets our SSL private key? In a certain sense, it turns HTTPS into HTTP: what was previously a confidential conversation between the two of us could be listened in by a third party.
Steps we have taken
As a first step, we’ve upgraded OpenSSL on all of our hosts, regardless of their exposure, and we’ve rotated all of our keys.
We also put pressure on AWS and were among the first companies to have our ELBs upgraded.
Knox, our PCI store which contains our card data, has also undergone full key
rotation, and we’ve re-encrypted all of our card data with a new key. You can
never be too safe.
We’re currently discussing things with our upstream vendors to make sure that they are also patched up quickly.
Doing a survey of our most commonly used client libraries, it seems that support for upgrading our client libraries to use certificates is pretty spotty, and requires manual intervention to verify that you’re not using a valid-but-revoked certificate.
We will need to upgrade our client libraries need to use the new certificate.
We are working on patches to add this functionality to our client libraries, but that will take some time. Expect an update on that front soon.
Steps you may wish to take
You can use publically available 3rd party code to check if your application stack is vulnerable.
Here are some helpful links:
As an extra precaution, we advise you to rotate your Balanced API keys. (and you probably should for your other services, too.) We’ve built a convenient GIF for you:
And that’s it!
If you have any questions about this vulnerabilty and how it affects you,
we’d be happy to help. Please jump into #balanced on
mahmoudimus, and we’d be happy to discuss
Every once in a while, a security issue happens. It’s unfortunate that this one is so serious and yet so widespread. We take these kinds of issues very seriously, and take steps as quickly as we can to protect you and your data.