Sunday, April 13, 2014

As the Heart Bleeds (A new cryptographic soap opera)


By now, I'm sure you've heard of HeartBleed.  If not, you've missed a fun time. A good description of what it's all about is at: http://blog.cryptographyengineering.com/2014/04/attack-of-week-openssl-heartbleed.html

One of the "interesting" aspects of this vulnerability is that it places the private SSL certificate for the server at risk.  At first, there was some doubt as to just how vulnerable the private certificate really was, but as of 4/12/2014, the vulnerability of a server's private certificate has been clearly demonstrated: http://arstechnica.com/security/2014/04/private-crypto-keys-are-accessible-to-heartbleed-hackers-new-data-shows/

The first person to publicly demonstrate that the private certificate is vulnerable, Fedor Indutny (https://twitter.com/indutny), also had a snippet of code on his twitter feed showing how to grab and view certificate revocation lists (CRL).  

I was idly curious ... since we now know that any server's private certificate has potentially been compromised ... how many folks have started to revoke their server's SSL certificates?

Below is the results of the hint provided by Indutny, and a bit of Perl.  The vertical axis is the number of certificates revoked on a given day, with time flowing from the left to the right.

Here we can see the number of certificates revoked over the past two years (see updates below for discussion of the spike around 4/16):



And here we can see the activity over the past month.  Since we know that some folks received advanced notification of the vulnerability, I was especially interested to know if there a spike in revocations prior to the announcement.  (Which I don't see.)



Just to provide some context, here's the same data for this calendar year. 


If you're interested in drilling into this a bit deeper, check out the numerous charts at:


Obviously, I'll keep an eye on certificate revocations for awhile.  All these charts are now being updated automatically every night.

It'll be interesting to see what the next weeks will bring.  Some folks are predicting a meltdown of both the CRL handling infrastructure (straight CRLs, OCSP and Google's CRLSet), as well as the infrastructure for issuing replacement SSL certificates.


Update (4/15/2014):  Those who know me know I'm extraordinarily lazy, I'll spend days getting a computer to do 5 minutes of work for me.  :-)  I've automated producing these charts, and through the magic of cron I'll be updating them daily.  They're not the pretty ones I created with Excel previously, but gnuplot is my friend in this case.


Update (4/16/2014): The good folks as SANS published a diary entry yesterday looking at the same issues.  They produced a similar chart, but from a much richer set of sources.  When I contacted them to ask who their sources were, not only did they provide a list - they put up a spiffy page which features an automated chart.  So the charts here now have a total of 16 different sources for CRLs.  Whoopee!


Here's a link to their CRL tracking page: https://isc.sans.edu/crls.html


Update (4/16/2014 - #2): SANS has provided an even more comprehensive list of CRLs.  Also they noticed that with the enhanced list, there's a massive spike in certificate revocations ... due to a massive number of certificate revocations from Globalsign.com.




Update (4/18/2014): The rest of the world is starting to catch up on this issue.  This article looks at the impact of massive CRL changes to the "performance" of the Internet - and confirms that Akamai is planning a major round of certificate revocations: http://www.zdnet.com/internet-slowed-by-heartbleed-identity-crisis-7000028506/

Here, BTW, is a observation from Cloudflare on the cost of massive certificate revocation: http://blog.cloudflare.com/the-hard-costs-of-heartbleed


Update (4/20/2014):  In an effort to provide more comprehensive monitoring, I've modified (actually rewritten) my program to provide a plethora of charts.  They're updated daily, and available at this URL:


I've added this link to the primary blog entry above. 


Update (9/10/2014): This excellent paper just came out.  A comprehensive review of how site admins responded to the Heat Bleed vulnerability.  Two Thumbs up!






And here's the updated list of the CRL sources which both SANS and I use:







  • http://corppki/crl/MSIT Machine Auth CA 2(1).crl,
  • http://crl-ssl.certificat2.com/keynectis/class2keynectisca.crl,
  • http://crl.comodoca.com/COMODOExtendedValidationSecureServerCA.crl,
  • http://crl.comodoca.com/COMODOHigh-AssuranceSecureServerCA.crl,
  • http://crl.comodoca.com/COMODOSSLCA.crl,
  • http://crl.entrust.net/level1c.crl,
  • http://crl.globalsign.com/gs/gsdomainvalg2.crl,
  • http://crl.globalsign.com/gs/gsorganizationvalg2.crl,
  • http://crl.godaddy.com/gdig2s1-42.crl,
  • http://crl.godaddy.com/gds1-54.crl,
  • http://crl.godaddy.com/gds1-85.crl,
  • http://crl.microsoft.com/pki/mscorp/crl/MSIT Machine Auth CA 2(1).crl,
  • http://crl.netsolssl.com/NetworkSolutions_CA.crl,
  • http://crl.omniroot.com/PublicSureServerSV.crl,
  • http://crl.startssl.com/crt2-crl.crl,
  • http://crl.usertrust.com/USERTrustLegacySecureServerCA.crl,
  • http://crl2.netsolssl.com/NetworkSolutions_CA.crl,
  • http://crl3.digicert.com/ca3-g27.crl,
  • http://crl3.digicert.com/sha2-ev-server-g1.crl,
  • http://crl3.digicert.com/ssca-g5.crl,
  • http://crl4.digicert.com/ca3-g27.crl,
  • http://crl4.digicert.com/sha2-ev-server-g1.crl,
  • http://EVIntl-crl.verisign.com/EVIntl2006.crl,
  • http://EVSecure-crl.verisign.com/EVSecure2006.crl,
  • http://gtssl-crl.geotrust.com/crls/gtssl.crl,
  • http://gtssl2-crl.geotrust.com/gtssl2.crl,
  • http://mscrl.microsoft.com/pki/mscorp/crl/MSIT Machine Auth CA 2(1).crl,
  • http://pki.google.com/GIAG2.crl,
  • http://sd.symcb.com/sd.crl,
  • http://svr-sgc-crl.thawte.com/ThawteSGCG2.crl,
  • http://SVRIntl-G3-crl.verisign.com/SVRIntlG3.crl,
  • http://SVRSecure-G3-crl.verisign.com/SVRSecureG3.crl