As the Heart Bleeds (A new cryptographic soap opera)

By now, I'm sure you've heard of HeartBleed.  If not, you've missed a fun time. A good description of what it's all about is at: http://blog.cryptographyengineering.com/2014/04/attack-of-week-openssl-heartbleed.html

One of the "interesting" aspects of this vulnerability is that it places the private SSL certificate for the server at risk.  At first, there was some doubt as to just how vulnerable the private certificate really was, but as of 4/12/2014, the vulnerability of a server's private certificate has been clearly demonstrated: http://arstechnica.com/security/2014/04/private-crypto-keys-are-accessible-to-heartbleed-hackers-new-data-shows/

The first person to publicly demonstrate that the private certificate is vulnerable, Fedor Indutny (https://twitter.com/indutny), also had a snippet of code on his twitter feed showing how to grab and view certificate revocation lists (CRL).  

I was idly curious ... since we now know that any server's private certificate has potentially been compromised ... how many folks have started to revoke their server's SSL certificates?

Below is the results of the hint provided by Indutny, and a bit of Perl.  The vertical axis is the number of certificates revoked on a given day, with time flowing from the left to the right.

Here we can see the number of certificates revoked over the past two years (see updates below for discussion of the spike around 4/16):

And here we can see the activity over the past month.  Since we know that some folks received advanced notification of the vulnerability, I was especially interested to know if there a spike in revocations prior to the announcement.  (Which I don't see.)

Just to provide some context, here's the same data for this calendar year. 

If you're interested in drilling into this a bit deeper, check out the numerous charts at:

http://del.thearmstrongfamily.org/images/List-Charts.html

Obviously, I'll keep an eye on certificate revocations for awhile.  All these charts are now being updated automatically every night.

It'll be interesting to see what the next weeks will bring.  Some folks are predicting a meltdown of both the CRL handling infrastructure (straight CRLs, OCSP and Google's CRLSet), as well as the infrastructure for issuing replacement SSL certificates.

---

Update (4/15/2014):  Those who know me know I'm extraordinarily lazy, I'll spend days getting a computer to do 5 minutes of work for me.  :-)  I've automated producing these charts, and through the magic of cron I'll be updating them daily.  They're not the pretty ones I created with Excel previously, but gnuplot is my friend in this case.

---

Update (4/16/2014): The good folks as SANS published a diary entry yesterday looking at the same issues.  They produced a similar chart, but from a much richer set of sources.  When I contacted them to ask who their sources were, not only did they provide a list - they put up a spiffy page which features an automated chart.  So the charts here now have a total of 16 different sources for CRLs.  Whoopee!

Here's the SANS diary entry: https://isc.sans.edu/forums/diary/INFOCon+Green+Heartbleed+-+on+the+mend/17961

Here's a link to their CRL tracking page: https://isc.sans.edu/crls.html

---

Update (4/16/2014 - #2): SANS has provided an even more comprehensive list of CRLs.  Also they noticed that with the enhanced list, there's a massive spike in certificate revocations ... due to a massive number of certificate revocations from Globalsign.com.

Here's a page where they document this spike: https://isc.sans.edu/forums/diary/Heartbleed+CRL+Activity+Spike+Finally+Found/17977

---

Update (4/18/2014): The rest of the world is starting to catch up on this issue.  This article looks at the impact of massive CRL changes to the "performance" of the Internet - and confirms that Akamai is planning a major round of certificate revocations: http://www.zdnet.com/internet-slowed-by-heartbleed-identity-crisis-7000028506/

Here, BTW, is a observation from Cloudflare on the cost of massive certificate revocation: http://blog.cloudflare.com/the-hard-costs-of-heartbleed

---

Update (4/20/2014):  In an effort to provide more comprehensive monitoring, I've modified (actually rewritten) my program to provide a plethora of charts.  They're updated daily, and available at this URL:

http://del.thearmstrongfamily.org/images/List-Charts.html

I've added this link to the primary blog entry above. 

---

Update (9/10/2014): This excellent paper just came out.  A comprehensive review of how site admins responded to the Heat Bleed vulnerability.  Two Thumbs up!

https://jhalderm.com/pub/papers/heartbleed-imc14.pdf

---

And here's the updated list of the CRL sources which both SANS and I use:

http://corppki/crl/MSIT Machine Auth CA 2(1).crl,

http://crl-ssl.certificat2.com/keynectis/class2keynectisca.crl,

http://crl.comodoca.com/COMODOExtendedValidationSecureServerCA.crl,

http://crl.comodoca.com/COMODOHigh-AssuranceSecureServerCA.crl,

http://crl.comodoca.com/COMODOSSLCA.crl,

http://crl.entrust.net/level1c.crl,

http://crl.globalsign.com/gs/gsdomainvalg2.crl,

http://crl.globalsign.com/gs/gsorganizationvalg2.crl,

http://crl.godaddy.com/gdig2s1-42.crl,

http://crl.godaddy.com/gds1-54.crl,

http://crl.godaddy.com/gds1-85.crl,

http://crl.microsoft.com/pki/mscorp/crl/MSIT Machine Auth CA 2(1).crl,

http://crl.netsolssl.com/NetworkSolutions_CA.crl,

http://crl.omniroot.com/PublicSureServerSV.crl,

http://crl.startssl.com/crt2-crl.crl,

http://crl.usertrust.com/USERTrustLegacySecureServerCA.crl,

http://crl2.netsolssl.com/NetworkSolutions_CA.crl,

http://crl3.digicert.com/ca3-g27.crl,

http://crl3.digicert.com/sha2-ev-server-g1.crl,

http://crl3.digicert.com/ssca-g5.crl,

http://crl4.digicert.com/ca3-g27.crl,

http://crl4.digicert.com/sha2-ev-server-g1.crl,

http://EVIntl-crl.verisign.com/EVIntl2006.crl,

http://EVSecure-crl.verisign.com/EVSecure2006.crl,

http://gtssl-crl.geotrust.com/crls/gtssl.crl,

http://gtssl2-crl.geotrust.com/gtssl2.crl,

http://mscrl.microsoft.com/pki/mscorp/crl/MSIT Machine Auth CA 2(1).crl,

http://pki.google.com/GIAG2.crl,

http://sd.symcb.com/sd.crl,

http://svr-sgc-crl.thawte.com/ThawteSGCG2.crl,

http://SVRIntl-G3-crl.verisign.com/SVRIntlG3.crl,

http://SVRSecure-G3-crl.verisign.com/SVRSecureG3.crl

Every Little Bit Helps

I thought this was a pretty interesting response to the debacle that our current certificate infrastructure has become.

https://sites.google.com/site/certificatetransparency/ev-ct-plan

In short, Google is going to try to encourage the use of Certificate Transparency to help deal with the weakness in our certificate infrastructure exposed by the recent rash of invalid certificate incidents.

Certificate Transparency is the idea that whenever a new certificate is issued, that event is logged in a public logfile.  In fact, anyone can log a certificate to a pubic logfile.  Interested folks could then audit the logfile for signs of fraud or erroneously issued certificates.

In other words, if users can reject certificates not published in the logfile, certificate forgers will have to publish their forged certificates in the public logfile for them to be useful ... permitting the legitimate domain owners to see that a forged certificate has been issued.

For a desciption of the concept, see:
https://sites.google.com/site/certificatetransparency/what-is-ct

On the one hand, I really like the idea.  As I've mentioned before, I firmly believe that the way to make the Internet more secure is to move aggressively to improve transparency and utilize open-source.

On the other hand, it really doesn't change anything fundamentally.  The issuers of certificates will still be the primary source of information in the logfile, which means that a rogue or compromised certificate authority will still be able to issue invalid certificates.  The logfile will just, potentially, provide a better opportunity for those who are paying attention to catch bad certificates more quickly.

There is an opportunity for Certificate Transparency to provide additional value, I think.  I'm no certificate expert, but what if when users are presented a certificate not listed in a logfile, in addition to having the option of rejecting the certificate they also "log" it to the logfile as suspect - providing in effect a distributed early warning system for questionable certificates.  In fact, I would imagine that the logfile service can simply track and publicize when it's queried for certificates which it doesn't know about (note: there are probably privacy issues with just automatically publicizing failed certificate lookups.)

Part of the challenge will be giving users the ability to query about certificates when they find themselves in a hostile environment.  Consider for example when a nation state controls DNS and is trying to use forged certificates to conduct man-in-the-middle attacks.  If the attacker can spoof or compromise a logfile service, then we're back to square one.

I still sometimes dream of a PGP style web-of-trust certificate system, which would rely on multiple links of trust used to generate a score based on how broadly trusted a given certificate is ... unlike the current hierarchical, centralized chain-of-trust we currently use.  The biggest problems with a web-of-trust is that it probably wouldn't scale well enough, and it does place more of the responsibility on the user to manage how they allocate trust.

In any event, it's always nice when elephantine companies like Google throw their weight behind something that can only help.

Certificate Owner Identity Theft

Here's yet more reason to believe that our digital certificate infrastructure is broken.

A CA named DigiCert has been issuing certificates for a company which went out of business in 2011. As recently as November 2012, they issued a certificate in the name of this defunct company - which not suprisingly was being used by malware designed for on-line banking fraud.

http://www.h-online.com/security/news/item/Certified-online-banking-trojan-in-the-wild-1808898.html

Reminds me of the (very) old trick of stealing the identity of somebody who died at an early age.  :-(