How-To: Download From Usenet

I was recently asked by a friend about how they could go about down­load­ing from Usenet in an effi­cient and rel­a­tively safe man­ner.  Rather than answer them directly, I fig­ured oth­ers out there might find this use­ful as well and posted it here.


The Big Pic­ture
Usenet has a large col­lec­tion of news­groups, some of which con­tain hun­dreds of thou­sands of binary arti­cles posted to them.  The con­nect to Usenet is made through a News­group Provider.  Exam­ples of these are GigaNews and EasyNews.  Once con­nected a per­son can browse through the news­groups man­u­ally with a news­group reader such as XNews.  The files can then be down­loaded to the user’s work­sta­tion and used.

Find­ing the Files
While man­ual brows­ing can be done, it’s much eas­ier to use a search por­tal such as Newzbin.  They have a pay site for most con­tent, and a free site for the XXX type stuff.  At about $0.50US a week, they aren’t a bad deal.  The pay­ment lets you down­load a NZB file.  This file is really just an XML doc­u­ment con­tain­ing infor­ma­tion about the loca­tion of the files needed.  Once this file is down­loaded it can be loaded into a pro­gram such as NZB-O-Matic.

The Con­nec­tion
I have used both EasyNews and GigaNews for many years.  Both are qual­ity providers, but I left EasyNews once GigaNews put out an unlim­ited plan.  I per­son­ally use their Dia­mond plan as it pro­vides not only unlim­ited band­width to their high speed servers, but it allows for encryp­tion of the con­nec­tion too.

The Sequence of Events

  1. Search for and down­load the desired .nzb file from Newzbin
  2. Load the file in NZB-O-Matic by using “Import NZB” under the “Trans­fer Queue” tab
  3. Click “Con­nect” in NZB-O-Matic under the “Usenet Servers” tab to start the transfer
  4. Once every­thing is down­loaded use Quick­Par to ver­ify the integrity of the files and repair dam­aged archives
  5. Extract the archives and enjoy your newly acquired files

Con­fig­ur­ing the Soft­ware
First of all, if using the recomended method of encryp­tion to GigaNews, make sure to install stun­nel and setup as described here.  Next, you’ll want to select “Add Server” in NZB-O-Matic pro­vid­ing it with “local­host” as the host­name, port 119, at least 10 con­nec­tions, and your new­group provider username/password. Your luck may vary, but with my com­cast con­nec­tion I’ve found that 10 con­nec­tions can sus­tain approx. 1000 KB/s down.

Graphs in Perl

It wasn’t until recently that I started to really use perl for my tasks. But the more I use it, the more I find myself enjoy­ing it’s sim­ple ele­gance and easy meth­ods for trans­form­ing large com­plex sets of data into some­thing mean­ing­ful. When­ever I find my self needed to ana­lyze logs, I like to see them in the con­text of over time. This is far from the only way to visu­al­ize data, but it’s a basic one that I find use­ful time and time again. GD::Graph makes this task insanely easy.

All that’s required for a basic graph is a two-dimensional array. Essen­tially, an array con­tain­ing two other arrays; one for the X-Axis val­ues, and another for the Y-Axis. The lit­tle code below cre­ates my data array, and sets the width/height of a new graph object.

@data = (\@xvalues,\@yvalues);
my $graph = GD::Graph::area->new($width,$height);

Now all I need to do is plot the data on the graph object and extract a pic­ture from it.

my $image = $graph->plot(\@data) or die $graph->error;
print STDOUT $image->png;

And that’s it!  A very easy and straight for­ward method to get a sim­ple graph of some data.

Spam Analysis

The other day I received a piece of spam that sur­prised me. First, because I was sim­ply shocked that I got a piece of spam. I haven’t received a sin­gle piece of spam at that address in years! This led to my sec­ond obser­va­tion: the spam was sent to one per­son and car­bon copied to three oth­ers, all at the same domain name. Now that’s tricky! Here lots of spam­mers are try­ing to use PDFs or images to get their spam through, and this per­son just made it look like a typ­i­cal busi­ness email.

SPAM Message

If you both­ered to take a look at the image above, you noticed a link to www [dot] med123window [dot] org. Now I’m always up for an adven­ture, so I logged into my trusty *nix box and pulled the page down with wget. Using cat to look at the con­tents, I noticed that it con­tained noth­ing but a big encoded JavaScript. Take a look at it here.

The script would use the JavaScript func­tion unescape to turn the hex val­ues into ASCII and then pass that result on to the eval func­tion for it to be loaded and exe­cuted at run­time. Maybe it’s because I look at mal­ware all day, but is it just me, or is there no good rea­son for eval to exist?

I could have used a JavaScript based sand­box approach that would allow the code to be unescaped and dis­played in a textarea, but I’ve heard reports that some mali­cious code is wise to that attempt and can defeat it. So I went with a method they can’t defeat. It’s called perl. And it’s beautiful!

The method I used here was to take input off of STDIN, find all occur­rences of \x??, con­vert each to the cor­re­spond­ing ASCII value, and then finally parse that result for use of HTML encod­ing that uses hex in the form %??. Here’s the result­ing pro­gram.
(** Note: In both exam­ples above the ques­tion marks would be replaced with a hex value 0–9 or A-F)

Here’s the out­put after both rounds of con­ver­sion through my script.

Spam Analysis - Round 2

Thank­fully, this tries to load a page off of non-standard port 8088 which is blocked by default in my gen­eral egress rule set. It’s blocked in yours too, right?

If we use a machine that does have access to the port we can view this site by pro­vid­ing a fake sub­do­main match­ing the pat­tern the script above would use. When I saw it I was actu­ally fairly impressed. The design and lay­out was very well done, looked pro­fes­sional, and dare I say even bet­ter than some legit cor­po­rate web sites!

The main site: Spam Analysis - Main Site The Check-Out Cart: Spam Analysis - Check Out

Now here’s what’s always baf­fled me about these sites.…. They are sell­ing drugs, right? So why are peo­ple will­ing to pay large sums of money for drugs when they could get from their doc­tor for a co-pay if they really needed them? Also, who says this dealer can be trusted?! They are an annoy­mous web­site that uses shad­ing mar­ket­ing tac­tics to get peo­ple to their site. What’s to say that the check out isn’t just to col­lect credit card num­bers and never send you a sin­gle pill? Even more dire, what’s to stop them from putting arsenic in every pill? It’s not like they are reg­u­lated by the FDA.

* shrugs * I may not under­stand why peo­ple go to these sites, but I can at least accept that some­one must, oth­er­wise there wouldn’t be so much effort put into get­ting the adver­tise­ment out.

College Classes and Security

As men­tioned in my About sec­tion, I’m cur­rently work­ing on my Mas­ter of Sci­ence in Com­puter Secu­rity. Fun­da­men­tally its only a hand­ful of classes dif­fer­ent from the MS in Com­puter Sci­ence so I’m tak­ing a slew of pro­gram­ming cen­tric courses.

This week I received the book for next semes­ter and started read­ing through it. I’m very happy to say that writ­ing code with secu­rity in mind was men­tioned sev­eral times. And it’s not all just your stan­dard stuff about buffer over­flows either. This selec­tion below is from the preface:

Hav­ing a solid under­stand­ing of com­puter arith­metic is crit­i­cal to writ­ing reli­able pro­grams. For exam­ple, one can­not replace the expres­sion (x<y) with (x-y<0) due to the pos­si­bil­ity of over­flow. One can­not even replace it with the expres­sion (-y<-x) due to the asym­met­ric range of neg­a­tive and pos­i­tive num­bers in thetwo’s com­pli­ment rep­re­sen­ta­tion. Arith­metic over­flow is com­mon source of pro­gram­ming errors, yet few other books cover the prop­er­ties of com­puter arith­metic from a programmer’s per­spec­tive.1

1 Bryant, Ran­dal E. and David R. O’Hallaron. Com­puter Sys­tems: A Programmer’s Per­spec­tive.
        New Jer­sey: Pear­son Edu­ca­tion, 2003.

Unblocking “Dangerous” Attachments in Outlook 2007

I recently came across an email that had an attach­ment of a very spe­cific file type. I was expect­ing this email and the attach­ment. Out­look how­ever, didn’t know about this file exten­sion and decided that it should be blocked. After some search­ing I came across this KB Arti­cle that describes how to reme­di­ate this issue in Out­look 2000.

Some minor tweaks and it works for Out­look 2007 too!

Name: Level1Remove
Type: REG_SZ
Data: List of file exten­sions sep­a­rated with a comma includ­ing the period. (Ex: .mdb,.xnk)

If you want to block exten­sions nor­mally allowed mod­ify Level1Add instead of Level1Remove.

Simple UNIX Time Conversion

Often when inves­ti­gat­ing inci­dents, log files, file for­mats, etc I come across a Date and Time stored in UNIX for­mat. That is, the num­ber of sec­onds since Mid­night on Jan­u­ary 1st, 1970. The script is very straight for­ward and easy to use. Pass it the UNIX time on the com­mand line and it will out­put the time in a human read­able format.

The Script

Sam­ple Out­put
Convert Time Output