Python Tip: Checking to see if your Python installation has support for SSL

I was trying to figure out if my installation of Python was compiled with SSL support and found it to be non-intuitive if you didn't compile Python for yourself.

So, to check if you have SSL support configured with your installation of Python, go to your command prompt and type:

python

and you'll get the Python interactive shell (that will look something like this):

Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

At the >>> prompt, type import socket:

Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket

Then check for the ssl attribute, by typing hasattr(socket, "ssl") at the >>> prompt and look for a True or False response:

Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> hasattr(socket, "ssl")
True
>>>

A True response means that SSL is compiled in your Python installation.

Good luck.

If you want some good books on learning to use Python, I highly recommend Beginning Python: From Novice to Professional by Magnus Hetland and Learning Python by Mark Lutz and David Ascher. I am currently using both books to get up to speed on the Python language and I am really enjoying working with both of them.

Bash Tip: Proving a Negative with grep and diff

I stumbled across an interesting problem a while back. Given a set of data how do you determine correlating data that isn't there?

I was given a file containing a list of names and some other lines that indicated a successful condition. If a name on one line was followed by a success statement, then the condition was successful for the previous line. However, if a name was followed by another name, then the condition had failed for the first name.

Confused already? Let's look at an example and see if that clears things up. Say we have a file, tally.txt, whose contents look something like this:

Doug
Jim
voteSuccess
Diana
voteSuccess
Thomas
voteSuccess
Drew
Elizabeth
Chris
Adrienne
voteSuccess
Nicholas
voteSuccess
Anita
Greg
Jacob
Trudy
voteSuccess
Alex
voteSuccess
Richard
voteSuccess
Donald
Sam
Steve
Bob
Nathan
voteSuccess
Penelope
voteSuccess
Bishop
voteSuccess
Dustin
voteSuccess
Ron
George
Henry
voteSuccess
Arthur
Reggie
voteSuccess

Here is the same file with line numbers added to make things clearer:

1 Doug
2 Jim
3 voteSuccess
4 Diana
5 voteSuccess
6 Thomas
7 voteSuccess
8 Drew
9 Elizabeth
10 Chris
11 Adrienne
12 voteSuccess
13 Nicholas
14 voteSuccess
15 Anita
16 Greg
17 Jacob
18 Trudy
19 voteSuccess
20 Alex
21 voteSuccess
22 Richard
23 voteSuccess
24 Donald
25 Sam
26 Steve
27 Bob
28 Nathan
29 voteSuccess
30 Penelope
31 voteSuccess
32 Bishop
33 voteSuccess
34 Dustin
35 voteSuccess
36 Ron
37 George
38 Henry
39 voteSuccess
40 Arthur
41 Reggie
42 voteSuccess

Lines 3, 5, 7, 12, 14, 19, 21, 23, 29, 31, 33, 35, 39, and 42 all indicate a success condition (voteSuccess). By the conventions of the file, that means that the people on the preceding lines actually had the success (lines 2, 4, 6, 11, 13, 18, 20, 22, 28, 30, 32, 34, 38, and respectively). The problem is that we want to find out who wasn't able to successfully vote. We need to find some way to extract those that voted successfully from the file and only leave those that weren't able to vote.

It should be noted that I simplified this example quite a bit. The success condition string (voteSuccess) could actually be one of a host of things, so it is not just one known string that we can work against, but it is good enough for this exercise.

Of course this whole situation would be a lot easier if the program that created this file placed some sort of indication of failure after the names of the people that didn't have success in voting. Unfortunately, in many instances, we're often stuck with the formats we're given and have to find a way to make them work.

After a little thought, I came up with a psuedo algorithm that I thought might solve the problem:

1. Correlate all the success conditions with the appropriate people.


2. Strip these into a file of successful voters sorted alphabetically.


3. Strip out all the status messages, leaving only voters, and sort them alphabetically into another file of all voters.


4. Check the difference between the files. As the successful voters will be in both files, only those that failed will be different.

For step one, we'll use a somewhat current version of the grep utility (the one I used was 2.5.1, you can find the version by typing grep -V) to print all the lines of tally.txt that contain voteSuccess and every line above them. The -B switch for grep prints however many lines you want above the string that you were looking for. Typing:

grep -B 1 voteSuccess tally.txt

You'll notice that the -B switch prints -- between contiguous blocks of matches.

Jim
voteSuccess
Diana
voteSuccess
Thomas
voteSuccess
--
Adrienne
voteSuccess
Nicholas
voteSuccess
--
Trudy
voteSuccess
Alex
voteSuccess
Richard
voteSuccess
--
Nathan
voteSuccess
Penelope
voteSuccess
Bishop
voteSuccess
Dustin
voteSuccess
--
Henry
voteSuccess
--
Reggie
voteSuccess

We'll clean those out by piping the output into an invert match of grep.

grep -B 1 voteSuccess tally.txt | grep -v ^[--]

Now we'll clean out the voteSuccess condition statements and sort the output.

grep -B 1 voteSuccess tally.txt | grep -v ^[--] | grep -v voteSuccess | sort

Our output from the first command sequence looks like this:

Adrienne
Alex
Bishop
Diana
Dustin
Henry
Jim
Nathan
Nicholas
Penelope
Reggie
Richard
Thomas
Trudy

Now that we have a list of the successful voters, let's redirect it to the file, successfulvoters.txt, that we'll later use to ferret out the failed voters.

grep -B 1 voteSuccess tally.txt | grep -v ^[--] | grep -v voteSuccess | sort  > successfulvoters.txt

Next, we need to pull together a sorted list of all voters. This is pretty easy, all we have to do is an inverted search for the term voteSuccess. The only things left will be the names of all the voters which we can sort and redirect into the file allvoters.txt

grep -v voteSuccess tally.txt | sort > allvoters.txt

Finally, we'll compares the successfulvoters.txt and allvoters.txt files using the diff utility. As diff can be verbose, we'll ask it to output an ed (line editor) script by employing the -e switch. These script instructions will highlight what needs to happen to the successfulvoters.txt file in order to make it look like the allvoters.txt file... which is add back all the failed users.

The failed users that we were trying to figure out how to isolate.

Let's compare files:

diff -e successfulvoters.txt allvoters.txt

That gives us the following:

12a
Ron
Sam
Steve
.
6a
Jacob
.
5a
Elizabeth
George
Greg
.
4a
Donald
Doug
Drew
.
3a
Bob
Chris
.
2a
Anita
Arthur
.

If you look at this output upside down, you can follow along in the successfullvoters.txt file and see where these additions would be added in order to make a complete list of users. If you can't do it mentally, I've flipped the output here:

.
Arthur
Anita
2a
.
Chris
Bob
3a
.
Drew
Doug
Donald
4a
.
Greg
George
Elizabeth
5a
.
Jacob
6a
.
Steve
Sam
Ron
12a

However, we just need the usernames and not the commands for the ed utility. If we get rid of every line that starts with a number (our voter names don't start with numbers) and every line with a period (.), and then run that through sort, we should have an alphabetical list of people that couldn't successfully vote.

To get rid of any line that begins for a number, we'll do an invert search on the output of the diff command. The expression ^[[:digit:]] uses the carat (^) character to denote starting the line and the shorthand expression [[:digit:]] to denote any number. That will just leave us to content with the periods, which we can remove by piping this output into yet another inverted grep search asking to return any line that doesn't contain them, [.]. Then we sort the output to make it more useable.

diff -e successfulvoters.txt allvoters.txt | grep -v ^[[:digit:]] | grep -v [.] | sort

And that's that!

We can put it all together in a quick and dirty bash script that will parse out the failed voters given a file name to process.

#!/bin/bash

# Take the filename from the command line and stuff it into a variable
TALLY=$1

# Find the successful voters
grep -B 1 voteSuccess $TALLY | grep -v ^[--] | grep -v voteSuccess | sort  > successfulvoters.txt

# Find all the voters
grep -v voteSuccess $TALLY | sort > allvoters.txt

# Find the difference between the successful voters
# and all the possible voters (ie the failed voters)
diff -e successfulvoters.txt allvoters.txt | grep -v ^[[:digit:]] | grep -v [.] | sort

Does anyone have other ideas how to tackle this problem? While the test case was relatively small, the actual data set contained tens of thousands of entries.

I see a lot of redundancy in this solution with its two passes over the tally file. On the other hand, I had a working solution in about 15 minutes.

I have tested this a couple of times and it looks to work on all my data sets. Perhaps there's a problem that I am not seeing and, if so, please speak up and let me know.

How would you solve this problem?

Do you have any file processing war stories? Tricks of the trade you'd like to share?

Also, I have a couple of texts that I have use to flesh out my understanding of scripting and the bash shell. The first is the O'Reilly book Learning the bash Shell by Cameron Newham. It is a step by step introduction to bash shell concepts and includes a good overview of many standard shell tools and techniques.

I also really like the more general book by Stephen Kochan and Patrick Wood titled Unix Shell Programming (3rd ed). Kochan and Wood write the book to the POSIX standard for shells which should help in writing maintainable and portable scripts, however they also make an effort to point out how each shell differs in its approach. It has its faults, as most books do, but it is solid nonetheless.

Bash Tip: Finding a Line and the One Following it

I was recently asked the following question:

I need to identify lines containing a string in a file and extract that line and
the next line from the file. There might be mutiple occurrences in the file.

In this example file I need to scan for "gottoget" and then extract line 3
and 4 as well as lines 6 and 7

Example file:
line 1
line 2
gottoget line 3
want this line as well line 4
line 5
gottoget line 6
ok this one must come with line 7
line 8
line 9
line 10

Hope you can help.

I think this puzzle is easily solved through the use of grep's -A flag.

According to the man page for grep (man grep), the -A flag prints the number of lines specified after the matching lines. It sounds like grep -A 1 gottotext examplefile should do the trick. This line will grab the line containing the string we're looking for ("gottotext" in your example) and the first line after that matching line. If we set grep up this way, we get the following:

gottoget line 3
want this line as well line 4
--
gottoget line 6
ok this one must come with  line 7

The -- line separates contiguous matches. If you don't want that, the lines are easily removed with another grep filter ( | grep -v ^[--]) which says to show everything but lines that begin with the -- characters. If you have -- characters that are legitimate at the beginning of some lines in your data, you may need to play around a bit to only filter out these unnecessary ones.

Putting it all together, we get:

grep -A 1 gottoget examplefile | grep -v ^[--]

Giving us the cleaned up output of:

gottoget line 3
want this line as well line 4
gottoget line 6
ok this one must come with  line 7

And that's it. A simple application of some grep statements provides the answer.

I highly recommend Unix Shell Programming (3rd edition) by Stephen Kochan and Patrick Wood if you are interested improving your understanding of shell scripting. Kochan and Wood do a very thorough job (using plenty of code examples) exploring various aspects of essential shell scripting tools and techniques.

Post a comment if you have a different or better way of handling this puzzle.

Take care.

Bash Tip: Reverse Sorting Lists in the Shell

Every once in a while I check my site logs and find a common search phrase in referrals from search engines. Often the visitors appear to leave immediately; presumably because the page they landed on didn't answer their question.

A phrase that has been appearing quite frequently lately is "bash reverse sort list".

I can't tell exactly what they mean by their search query, so we'll take a couple of cracks at it.

My first thought is that they might be looking to reverse the output of the command line tool ls.

Say we have a directory and we see the following files when we run ls:

a  aa  ab  b  bd  be  c  cqw  cw  d  de  defg  h  hi  hij  hijk  i  ii  iii  ij

To get a simple reverse listing of those files, we should use the -r switch for ls. typing ls -r in the same directory yields:

ij  iii  ii  i  hijk  hij  hi  h  defg  de  d  cw  cqw  c  be  bd  b  ab  aa  a

Having your file list reversed in a horizontal line isn't always useful when you are looking for a vertical list. It just takes a little bit of extra work to turn your list on its side if that's what you need.

First, we'll use the -l switch of ls to show the long listing for the files. Typing ls -lr gives us a reverse listing of our files in a vertical format.

total 0
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 ij
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 iii
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 ii
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 i
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 hijk
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 hij
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 hi
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 h
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 defg
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 de
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 d
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 cw
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 cqw
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 c
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 be
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 bd
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 b
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 ab
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 aa
-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 a

That's a bit more verbose than what I expect we're looking for, so we'll need to employ a couple of more tools to trim away the fat.

All we want is the last column (8, if you consider the columns delimited by spaces) of information. This is where the cut command comes in handy. It does exactly what the name implies by slicing and dicing data in multiple handy ways.

By default, the cut command treats data as fields separated by tabs. By sending the output of our ls -lr command as input to the cut command while changing the default delimiter character with the -d switch, we can filter out all but the 8th column. So far our command looks like this, ls -lr | cut -d" " -f8, and our ouput looks like this:

ij
iii
ii
i
hijk
hij
hi
h
defg
de
d
cw
cqw
c
be
bd
b
ab
aa
a

Almost perfect. However, you'll notice one small problem at the top of the list. There's an extra blank. If you look at the original output of ls -lr, it's quickly becomes clear where the blank line came from. The total 0 line in the original output had only two fields, total and 0, leaving nothing but a blank when cut went looking for the eighth field.

It's not too difficult a job to clean this up with a little creative application of the grep command. We'll use the -v or inverse match switch of grep (otherwise known as "show me everything but") of a line with only a beginning, represented by the carat (^) symbol, and an end, represented by the dollar sign ($) and nothing in between, or -v ^$.

Putting it all together as ls -lr | cut -d" " -f8 | grep -v ^$ successfully removes the blank line from our vertical reverse sorted list of files.

ij
iii
ii
i
hijk
hij
hi
h
defg
de
d
cw
cqw
c
be
bd
b
ab
aa
a

Another list you might like to sort is one contained in a file. ls isn't going to help us with this one, but the sort command is here to help.

By default, sort will sort a list in a file by the first field as delimited by white and non-white space. Taking an example file (sort.txt) containing the following:

a
b
bd
hij
be
aa
cqw
ab
c
cw
d
de
iii
defg
h
hi
hijk
i
ii
ij

So, running sort against sort.txt results in:

a
aa
ab
b
bd
be
c
cqw
cw
d
de
defg
h
hi
hij
hijk
i
ii
iii
ij

The sort command also offers a reverse sort option through the -r switch. Running sort -r against sort.txt (sort -r sort.txt) results in:

ij
iii
ii
i
hijk
hij
hi
h
defg
de
d
cw
cqw
c
be
bd
b
ab
aa
a

I hope this answers some of the basic questions about reverse sorting lists. For more information check out the manual pages for ls and sort (man ls and man sort).

However, you might just want your list flipped on its head, with no sorting whatsoever. Say you have the list:

a
d
c
b

You want it like to look like this:

b
c
d
a

As it turns out, there is a command just for that purpose called tac. Where cat will concatenate the contents of a file to the screen (standard output), tac will do the same after reversing the contents of a file.

Take the text of the 1st Amendment to the US Constitution, for example.

Congress shall make no law respecting an establishment of religion,
or prohibiting the free exercise thereof;
or abridging the freedom of speech,
or of the press;
or the right of the people peaceably to assemble,
and to petition the Government for a redress of grievances.

Running tac against these lines compeletely reverses them:

and to petition the Government for a redress of grievances.
or the right of the people peaceably to assemble,
or of the press;
or abridging the freedom of speech,
or prohibiting the free exercise thereof;
Congress shall make no law respecting an establishment of religion,

Whereas if we had used sort, the output would look slightly different:

and to petition the Government for a redress of grievances.
Congress shall make no law respecting an establishment of religion,
or abridging the freedom of speech,
or of the press;
or prohibiting the free exercise thereof;
or the right of the people peaceably to assemble,

If your list isn't vertical with items separated by a newline, you can use tac's -s switch, similar to cut's -d switch, to identify a different separator.

Update 1: A helpful reader pointed out that the ls examples could be a lot smaller with the application of the -1 switch to the ls command. This switch tells the standard ls command to print one file per line. When combined with the reverse, -r, switch, we get a reverse list of files in a vertical as opposed to the standard horizontal layout.

In the end, just typing

ls -1r

will result in this list of files

a  aa  ab  b  bd  be  c  cqw  cw  d  de  defg  h  hi  hij  hijk  i  ii  iii  ij

being printed like this

ij
iii
ii
i
hijk
hij
hi
h
defg
de
d
cw
cqw
c
be
bd
b
ab
aa
a

Update 2: It turns out that I didn't cover how to reverse sort a horizontal line. Since it is a little long, you can check out my solution in this post, Bash Tip: Reverse Sorting Lists Revisted; Reversing a Horizontal List.

--

I hope these tips help everyone out. If you want more resources on shell scripting, I highly recommend Unix Shell Programming (3rd edition) by Stephen Kochan and Patrick Wood if you are interested improving your understanding of shell scripting. Kochan and Wood do a very thorough job (using plenty of code examples) exploring various aspects of essential shell scripting tools and techniques

Bash Tip: Extracting a Range of Lines with sed

A little puzzle came across my desk the other day. I was asked how we could pull a range of lines from a file.

Say the file has ten lines:

Line 1
Line 2
Line 3
Line 4
Line 5
Line 6
Line 7
Line 8
Line 9
Line 10

We want to extract only lines five through eight from the file.

This isn't a hard problem, and there are probably a thousand ways to do this. However, I chose to use a simple sed command.

By typing the following we can get the result we want.

sed -n '5,8p;8q' filename

This line basically tells sed to start at line 5 and print until line 8 and then end processing at line 8. This simple construct works equally well with very large files and wider ranges of lines. Your output will look something like this:

Line 5
Line 6
Line 7
Line 8

It also works for extracting single lines if you change the line to look something like this:

sed -n '5p;5q' filename

The above line prints the single line (in this case "Line 5") and then stops processing the file.

I recommend the books Classic Shell Scripting by Arnold Robbins and Nelson H.F. Beebe and O'Reilly's Sed and Awk (second edition) by Dale Dougherty and Arnold Robbins if you would like additional resources for learning how to effectively leverage the sed utility.

How do you extract lines from a file? What tools and techniques work for you?

Komodo Tip: Launching a Python Shell and Running Code from Komodo Edit 4.2 on Linux

I have been teaching myself python lately and as part of that I have been trying out a few editors including vim, geany, and (most recently) Komodo Edit 4.2 from the folks at Active State (makers of fine cross platform scripting tools and language ports).

Komodo Edit appears to be a stripped down version of Active States more full featured Komodo IDE. Even though it is stripped down, it still has a some nice features.

While the Komodo Edit doesn't have all the built in features of the full-fledged IDE, the inclusion of the extensible Toolbox allows you to approximate some tools and integration that turns out to be quite handy.

Launching a Python Shell

The Komodo IDE has a nice integrated shell for python that makes trying out code quite easy. This is one of the features that was left out of Komodo Edit, but we can make launching a shell a little easier by creating a Toolbox entry that you can then bind to a key combination and launch from within the editor.

To get the entry:

1. Launch Komodo Edit
2. Click the Toolbox drop down menu
3. Choose the Add menu item
4. Then, click the New Command item
5. In the Add Command window, type Python Shell in the first feild
6. Then type, gnome-terminal -x python (or your terminal emulator of choice with the appropriate execute switch)
7. The choose, No Console (GUI Application) from the Run In: field
8. Next, click the Key Binding tab at the top and assign a keyboard combo that you can use to easily launch a python shell.

Running a Python File

Similarly you can create a Run Command to use python to execute the current file that you are working with.

To get the entry:

1. Launch Komodo Edit
2. Click the Toolbox drop down menu
3. Choose the Add menu item
4. Then, click the New Command item
5. In the Add Command window, type Run Python File in the first feild
6. Then type, %(python) %F
7. The choose, Command Output Tab from the Run In: field
8. Next, click the Key Binding tab at the top and assign a keyboard combo that you can use to easily launch a python shell.

Make sure to save the file before launching the Run Python File tool. If you are successful, your code will appear in a window at the bottom of the editor. If you get a stack trace, you can double click on the error and your cursor will jump back to the editor window to the line that is causing the problem.

I will post any more tricks in Komodo Edit as I come across them. If you have any tips or tricks, please post them to the comments.


CC photo credit: Fred Hsu