tag:blogger.com,1999:blog-88493237137878088582024-03-12T15:58:13.127-07:00Morlock HeadquartersJust another Morlock in TrainingJimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-8849323713787808858.post-51566731213818080182008-09-25T11:52:00.000-07:002009-08-20T10:02:28.951-07:00Mike MacCana’s VentureCake - The 5 lines that mystified O’Reilly - how to use a spreadsheet in bashYou should read <span style="font-weight: bold;">Mike MacCana’s post "The 5 lines that mystified O’Reilly: how to use a spreadsheet in bash"</span>. <span style="color: rgb(204, 0, 0);">NOTE: I took down the link as it is now parked. Venture Cake appears to be gone.</span><br /><br />Given that I just <a href="http://morlockhq.blogspot.com/2008/08/book-review-bash-cookbook.html">reviewed the Bash Cookbook</a> mentioned in Mike's post, I was really happy to find this elegant solution to the problem of processing comma separated value (CSV) files with a simple bash script which doesn't appear in its pages.<br /><br />It isn't an all purpose solution by any means, but a useful one if you need to expand on it. Some discussion of its finer good points and failings can be found on the <a href="http://news.ycombinator.com/item?id=313733">Hacker News post</a> that alerted me to it in the first place.Jimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com4tag:blogger.com,1999:blog-8849323713787808858.post-36168526713889511172008-09-23T11:33:00.001-07:002008-09-24T07:16:26.980-07:00Bash Tip: Reverse Sorting Lists Revisted; Reversing a Horizontal List<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6pY_ue3pbPTJ8absvT5cNT_fRu9dOVkN7kM03jLNAbiqqVyip_R4EUvT8ZHU6UNAx3aTtRcgm3LrfpZsBD1ffusGVTjMxk8prwKL8308PIaLOAvTO01BNwzSJ3s6ZhGdzLsS51t9uc14/s1600-h/swimming-backwards.jpg"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6pY_ue3pbPTJ8absvT5cNT_fRu9dOVkN7kM03jLNAbiqqVyip_R4EUvT8ZHU6UNAx3aTtRcgm3LrfpZsBD1ffusGVTjMxk8prwKL8308PIaLOAvTO01BNwzSJ3s6ZhGdzLsS51t9uc14/s320/swimming-backwards.jpg" alt="" id="BLOGGER_PHOTO_ID_5249286964548795378" border="0" /></a>A while back, I noticed that I was getting a lot of hits to my site from search engines like Google with search terms containing the terms "<span style="font-style: italic;">bash reverse sort list</span>." So I decided to write <a href="http://morlockhq.blogspot.com/2008/04/bash-tip-reverse-sorting-lists-in-shell.html">a post about various ways to reverse a list</a> using bash's tools. If my hit counter is any indicator, it is the most popular post I have written yet with thirty out of the last one hundred hits entering on that page.<br /><br />Recently, I was messing around with a data problem and realized that I needed to reverse a horizontal list. "No problem," I thought, "I'll just use one of those techniques I talked about before." Unfortunately, I didn't cover that case.<br /><br />On Linux, this is pretty simple, I found, if you have access to the <span style="font-style: italic;">rev</span> command.<br /><br />Say we have input in the form of the variable named <span style="font-style: italic;">testingHZsort</span> that looks something like this:<br /><br /><pre><br />bash $ testingHZsort="a aa ab b bd be c cqw cw d de defg h hi hij hijk i ii iii ij"<br />bash $<br />bash $ echo ${testingHZsort}<br />a aa ab b bd be c cqw cw d de defg h hi hij hijk i ii iii ij<br />bash $<br /></pre><br /><br />Now, because of variable expansion, we wouldn't be able to call <span style="font-style: italic;">rev</span> on the variable directly as we'll get a bunch of <span style="font-style: italic;">"No such file or directory"</span> errors as rev tries to find files by the names in individual elements of <span style="font-style: italic;">testingHZsort</span>. To get around this problem, we just have to expand the variable and then pipe it to the <span style="font-style: italic;">rev</span> command.<br /><br /><pre><br />bash $ echo ${testingHZsort} | rev<br />ji iii ii i kjih jih ih h gfed ed d wc wqc c eb db b ba aa a<br /></pre><br /><br />Works like a charm.<br /><br />The only problem is that you have to have access to the <span style="font-style: italic;">rev</span> utility, which isn't available on the versions of Solaris that I have access to, so I had to go back to the drawing board.<br /><br />On Solaris, I do have access to the <span style="font-style: italic;">sort</span> utility, which works fine on vertical lists and include a reverse (<span style="font-style: italic;">-r</span>) option. All I have to do is convert my vertical data into horizontal data, execute the reverse sort and then convert the vertical output back to horizontal output. Piping a couple of common utilities together should produce the desired result.<br /><br />The first thing to do is convert the horizontal list into a vertical list, we can do this using the <span style="font-style: italic;">tr</span>, or translate, utility to convert the spaces (' ') between testingHZsort's elements into newlines ('\n').<br /><br /><pre><br />bash $ echo ${testingHZsort}<br />a aa ab b bd be c cqw cw d de defg h hi hij hijk i ii iii i<br />bash $<br />bash $ echo ${testingHZsort} | tr ' ' '\n'<br />a<br />aa<br />ab<br />b<br />bd<br />be<br />c<br />cqw<br />cw<br />d<br />de<br />defg<br />h<br />hi<br />hij<br />hijk<br />i<br />ii<br />iii<br />ij<br />bash $<br /></pre><br /><br />Now we can pipe that output through <span style="font-style: italic;">sort</span> with the reverse (<span style="font-style: italic;">-r</span>) option.<br /><br /><pre><br />bash $ echo ${testingHZsort} | tr ' ' '\n' | sort -r<br />ij<br />iii<br />ii<br />i<br />hijk<br />hij<br />hi<br />h<br />defg<br />de<br />d<br />cw<br />cqw<br />c<br />be<br />bd<br />b<br />ab<br />aa<br />a<br />bash $<br /></pre><br /><br />Then we reverse the previous translation and replace all the newlines (<span style="font-style: italic;">\n</span>) with spaces (<span style="font-style: italic;">' '</span>).<br /><br /><pre><br />bash $ echo ${testingHZsort} | tr ' ' '\n' | sort -r | tr '\n' ' '<br />ij iii ii i hijk hij hi h defg de d cw cqw c be bd b ab aa a bash $<br /></pre><br /><br />That leaves us with a little problem of a final newline being converted into a space and not return our bash prompt (<span style="font-style: italic;">bash $</span>) to its proper position. We can fix that by appending a final newline to the output with <span style="font-style: italic;">printf</span>.<br /><br /><pre><br />bash $ echo ${testingHZsort} | tr ' ' '\n' | sort -r | tr '\n' ' '; printf "\n"<br />ij iii ii i hijk hij hi h defg de d cw cqw c be bd b ab aa a<br />bash $<br /></pre><br /><br />What about horizontal lines of data that are separated by some other delimiter than a space? What about a comma separated list?<br /><br />That's easy, actually. It is just a little different from the earlier examples in that we are going to replace the translation of the space into newlines with the translation of your new delimiter with newlines instead.<br /><br />So, let's say that <span style="font-style:italic;">testingHZLine</span> looks something like this:<br /><br /><pre><br />bash $ testingHZsort="a,aa,ab,b,bd,be,c,cqw,cw,d,de,defg,h,hi,hij,hijk,i,ii,iii,ij"<br />bash $ echo ${testingHZsort}<br />a,aa,ab,b,bd,be,c,cqw,cw,d,de,defg,h,hi,hij,hijk,i,ii,iii,ij<br />bash $<br /></pre><br /><br />We'll do the same sorting that we did before, but this time we'll replace all the commas (<span style="font-style:italic;">,</span>) with newlines (<span style="font-style:italic;">\n</span>).<br /><br /><pre><br />bash $ echo ${testingHZsort} | tr ',' '\n' | sort -r | tr '\n' ','<br />ij,iii,ii,i,hijk,hij,hi,h,defg,de,d,cw,cqw,c,be,bd,b,ab,aa,a,bash $<br /></pre><br /><br />We've lost our last newline again <span style="font-style:italic;">and</span> we have an extra comma to deal with. <br /><br />That's simple to clean up with a <span style="font-style:italic;">sed</span> substitution that will turn only the trailing comma into a newline character. We'll use the end of line positional regex character (<span style="font-style:italic;">$</span>) to do that. So we match a comma at the end of the line (<span style="font-style:italic;">,$</span>) and replace it with a newline (<span style="font-style:italic;">\n</span>).<br /><br /><pre><br />bash $ echo ${testingHZsort} | tr ',' '\n' | sort -r | tr '\n' ',' | sed -e 's/,$/\n/'<br />ij,iii,ii,i,hijk,hij,hi,h,defg,de,d,cw,cqw,c,be,bd,b,ab,aa,a<br />bash $<br /></pre> <br /><br />That's all there is to it.<br /><br />If you would like to improve your bash scripting skills you might want to consider picking up a copy of the <a href="http://www.amazon.com/exec/obidos/ASIN/0596526784/babytoolkit-20">Bash Cookbook</a>. I highly recommend it. You can read my full review of it, <a href="http://morlockhq.blogspot.com/2008/08/book-review-bash-cookbook.html">here</a>.<br /><br /><span style="font-size:78%;">cc photo credit: <a href="http://flickr.com/people/39904966@N00/">Man vyi</a></span>Jimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com1tag:blogger.com,1999:blog-8849323713787808858.post-13515224581219030412008-08-15T10:43:00.000-07:002008-12-01T06:55:13.793-08:00Book Review: The Bash Cookbook<span style="font-weight: bold;font-size:130%;" >The Elevator Pitch</span><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://www.amazon.com/exec/obidos/ASIN/0596526784/babytoolkit-20"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjybvja6VmiqcRs5vSfjZ0TVotuCbf-QK1z4BFCyuDxv3u7FalrYsVvRrYGJ7TIOrke6NWE2khFCGGjo3rWDRX2Vuhn8FRI3ecbll3dKDMGgk3bvEz0_GBnEBqYvIg7QokVXqj4W113Guw/s400/bashcookbook.gif" alt="" id="BLOGGER_PHOTO_ID_5239308469400777058" border="0" /></a><br />The <a href="http://www.amazon.com/exec/obidos/ASIN/0596526784/babytoolkit-20">Bash Cookbook</a> is a must own book for anyone that uses Unix and Linux for fun or profit. Bash is a powerful shell environment available in everything from Mac OS X to commercial Unix offerings like Solaris. Being comfortable and productive with this shell is going to make your life a helluva lot easier. The Bash Cookbook serves as a digestible tutor to this powerful shell while maintaining a depth that makes it a valuable reference for solutions to many of the common problems that command line power users face.<br /><br /><span style="font-weight: bold;font-size:130%;" >The Full Review</span><br /><br />I've been a Unix user since my first days studying Computer Science at college in the early nineties. While coming from using MS-DOS in high school and being plunked in front of a terminal with a dollar ($) prompt probably wasn't as disorienting as a move from Windows might have been, it was still pretty confusing. I struggled through the first few years until I took a systems programming class and finally started to understand the big picture of Unix. Still, it wasn't until almost a decade later that I decided to really try to wrap my mind around the Unix command line, and more specifically the <a href="http://en.wikipedia.org/wiki/Bash">bash</a> shell.<br /><br />As a Unix administrator, I have now been using the shell environment professionally for over five years. Bash is my shell of choice and I use it to do everything from processing various system logs, to running assorted backups, to creating system monitors, to wrapping more complex commands into usable interfaces, to transforming data into more usable formats. To get to that point, I spent a lot of time reading books like <a href="http://www.amazon.com/exec/obidos/ASIN/0596009658/babytoolkit-20">Learning the Bash Shell</a>, hanging out on the <a href="http://www.unix.com/shell-programming-scripting/">shell scripting forums at Unix.com</a> and reading <a href="http://planetsysadmin.com/">various sysadmin blogs</a>. All that is to say that I think I have a good grasp of the Unix/Linux command line in general and the bash shell in particular.<br /><br />Recently, I had the opportunity to read the <a href="http://www.amazon.com/exec/obidos/ASIN/0596526784/babytoolkit-20">Bash Cookbook</a>. Of all the technical books that I read for personal and professional gain, I prefer the formats of both <a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=oreilly%20hacks&tag=babytoolkit-20&index=books&linkCode=ur2&camp=1789&creative=9325">O'Reilly's Hacks series</a><img src="http://www.assoc-amazon.com/e/ir?t=babytoolkit-20&l=ur2&o=1" alt="" style="border: medium none ! important; margin: 0px ! important;" width="1" border="0" height="1" /> and its <a href="http://www.amazon.com/gp/search?ie=UTF8&keywords=oreilly%20cookbook&tag=babytoolkit-20&index=books&linkCode=ur2&camp=1789&creative=9325">Cookbooks</a><img src="http://www.assoc-amazon.com/e/ir?t=babytoolkit-20&l=ur2&o=1" alt="" style="border: medium none ! important; margin: 0px ! important;" width="1" border="0" height="1" /> for how they cover common problems and solutions in various technical subjects. I find them easy to digest, as both formats generally break large technical topics into bite sized chunks that present problems and solutions in very thorough, but approachable, ways. I find myself flying though these books. After reading a couple of pages that cover a single hack or recipe, I generally feel like I have learned something versus having to slog through twenty or so chapter pages in a typical tech book.<br /><br />Thankfully, The Bash Cookbook stands up with its predecessors. The authors, <a href="http://www.carlalbing.com/">Carl Albing</a>, <a href="http://www.jpsdomain.org/">JP Vossen</a> and Cameron Newham (also an author of the aforementioned Learning the Bash Shell) have backgrounds ranging from general technologists and authors to software developers for the Cray supercomputer company. This multifaceted experience set serves them well as they tackle various bash scripting topics from the mundane to the puzzling to the downright arcane. All of this is done with an approachable style and format that first identifies a problem, then offers a generalized solution, and finally follows up with a detailed discussion of the problem and solutions. This approach helps identify both the reasoning behind their solutions and the corner cases that will either further inform your own implementations or warn you that <span style="font-style: italic; font-weight: bold;">here be dragons</span>.<br /><br />The Bash Cookbook is divided into nineteen chapters and five appendixes, a few of which (most notably "Appendix D: Revision Control") could have served as full-on chapters by themselves. Topics include getting started with bash on various platforms (chapter 1); dealing with the intricacies of standard input and output redirection (chapters 2 and 3); job control (chapter 4); shell variables and arithmetic (chapters 5 and 6); finding and manipulating data (chapters 7, 8, and 9); working with functions and trapping conditions (chapter 10); manipulating dates and time (chapter 11); wrapping complex tasks (chapter 12); parsing files (chapter 13); writing scripts securely (chapter 14) ; bash corner cashes (chapter 15); customizing the bash environment (chapter 16); common system administration tasks (chapter 17); bash tips to be more productive (chapter 18); and, finally, common traps and workarounds for novice bash scripters (chapter 19). As you can see, there is a wealth of information to be had between the covers of this book.<br /><br />I found useful information from the beginning chapters (which are often throw away generalized instructions for getting up to speed in most tech books) all the way to the appendices themselves. Some standout recipes from the book include:<br /><ul><li>3.7 Selecting from a List of Options</li><li>5.2 Embedding Documentation in Shell Scripts</li><li>5.17 Giving an Error Message for Unset Parameters</li><li>5.19 Using Array Variables<br /></li><li>7.15 Showing Data As a Quick and Easy Histogram</li><li>8.3 Sorting IP Addresses</li><li>9.9 Finding Files by Content</li><li>10.6 Trapping Interrupts</li><li>13.4 Parsing Output into an Array</li><li>13.12 Isolating Specific Fields in Data</li><li>15.10 Finding My IP Address</li><li>15.13 Working Around "argument list too long" Errors</li><li>15.15 Sending Email from Your Script</li><li>16.4 Change your $PATH Temporarily</li><li>17.1 Renaming Many Files</li><li>17.8 Capturing File Metadata for Recovery</li><li>17.13 Prepending Data to a File</li><li>17.16 Finding Lines in One File But Not in the Other</li><li>17.17 Keeping the Most Recent N Objects</li><li>19.11 Seeing Odd Behavior from printf</li></ul>Chapter 14 of The Bash Cookbook demands special mention in this review. Titled "Writing Secure Shell Scripts", it opens with a general discussion of the need for writing secure shell scripts and gives a basic template utilizing many of the features that can make the average shell script more secure. The subsequent twenty three recipes flesh out this template with surprising, but approachable, detail. Of all the subjects in this book, this chapter's topic makes it worth buying and retaining as a goto reference. After skimming the recipes of chapter fourteen, I found a number of ways to make my scripts better. For instance, much of the "common wisdom" for creating temporary files that I have found around the Internet and in various books is simply wrong and, as chapter fourteen lays out, highly susceptible to race conditions. Some day, out of curiosity, I'd like to survey some open source projects that make use of shell scripts for installation, configuration, and maintenance and see how their handling of temporary files matches up. From the cursory searches that I have made, I am afraid what the results might show.<br /><br />The writing in the Cookbook is clear and to the point and incredibly consistent given that it was written by three writers. This is either a testament to the writing team and their ability to assimilate each other's styles or to O'Reilly's editorial staff's ability to tie the whole thing together (or, I assume, both). I particularly enjoyed the in depth discussion that many recipes received. It had the feel of looking over the shoulder of a veteran Unix admin and having the chance to pick his brain about why he was making the choices he was and why he was going about his business in a particular way. That is the book's greatest strength. As someone who has had to pick up Unix and Linux skills largely on his own, I found this approach invaluable. If you aren't surrounded by a Unix culture, it can be hard to pick up some of the more useful, but more complex, tricks of the trade. Think of The Bash Cookbook as your grey beard Unix hacker mentor on a shelf.<br /><br />The book and its Table of Contents and Index are so comprehensive with regards to the common types of tasks that one generally performs while writing shell scripts, that it has become, in the short time that I have had it, my first (and usually last) goto reference. If I forget how to search for keywords in files across directories for instance, it just took a quick scan of the Index to find a very good and working answer. I use this book so much, that I am considering buying a second copy to keep at home so I don't have to haul my dog-eared version back and forth to and from work. It is <span style="font-style: italic;">that</span> useful.<br /><br />Really.<br /><span style="font-weight: bold;"></span><span style="font-weight: bold;font-size:130%;" ><br />Some Nits to Pick<br /></span><br />As with any large project such as a book, there are bound to be a few things that slip through the cracks. The Bash Cookbook is no different. For instance, recipe 6.6 talks about the different ways to check for equality in bash including the use of the single equals (=) or double equals (==) signs. Functionally these two constructs are exactly the same, but using the single equals is more portable as it follows the POSIX standard. That's fine, and very good to know. However, the use of these constructs isn't consistent in the book, which could lead to confusion as the explanation that it really doesn't matter doesn't happen until page sixty four. Even worse, much earlier in the book recipe 3.7 is an example of the use of these two constructs not even being consistent in a single script where the variable $directory is checked for equality with the string "Finished" on one line with the double equals construct (==) and another with the single equals construct (=). From a script maintainability respect, being this inconsistency is a bad idea.<br /><br />One problem is the seeming omission of the treatment of arrays in bash. Most people unfamiliar with bash don't even realize that there are simple single dimensional arrays available in the environment, so I was happy to see some recipes that covered this topic. However, some of the more powerful array manipulations techniques, such as the ability to find the number of elements in an array with the simple ${array_name[@]} construct or the length of an individual array item with the ${#array_name[index number]} construct which is covered in the discussion of recipe 13.4, "Parsing Output into an Array", are buried in other recipes and hard to find even with the index. This misunderstanding could probably be helped if the See Also sections of each recipe pointed to other recipes in the book that dealt with similar subjects. Recipe 5.19, "Using Array Variables," only points to a section the the O'Reilly book Learning the Bash Shell. Other recipes in the book do a fine job of pointing out external sources of information as well as other recipes, so I think this is just a matter of some editorial consistency that would need to be beefed up for the next edition.<br /><br />The authors make a conscious effort to stick with core bash tools throughout the text. As the note in the "Preface" of the book, Perl is covered elsewhere. Though they do say they are okay using the right tool for the job and sometimes they tell you when it is best to use something else... much better than having the reader beat their heads against a wall in my opinion. This is a book about bash after all and it would be maddening if many solutions switched to other non-bash solutions whenever something wasn't readily able to be solved with the bash tool set. Unfortunately there are times when they overlook common bash tools in favor of other scripting languages like sed and awk. The prime example of recipe 7.10, "Keeping Some Output, Discarding the Rest" where they use awk to solve the problem. Sure, awk works, but I would have preferred if they would have at least mentioned the cut utility in this context if only for comparison sake. They should have at least linked to recipe 8.4, "Cutting Out Parts of Your Output", and recipe 13.12, "Isolating Specific Fields in Data." Again, I this is just a matter of editorial consistency and just one of only a few examples where the fact that the book was written by multiple authors becomes mildly apparent.<br /><br />Some other minor editorial issues revolve around typos and other minor errata. On page 84, "Thought" in the third <span style="font-style: italic;">Discussion</span> paragraph should be "Though." On page 64, the comment (after the #) at the end of the script states "end of while not finished" which can be confusing as the loop construct is actually an <span style="font-style: italic;">until</span> statement. Page 207 should be "what" instead of "hat" in the first full sentence of the page and similarly "fpllowing" on page 233 should be "following." For a book this long (622 pages), that's not bad at all. There may be others, but they weren't obvious during my reading. For a technical book, in its first edition, I was happy with the overall quality of the material.<br /><br />My final suggestion is for the inclusion of sample input and outputs for the scripts. Many scripts give these types of examples, which makes it endlessly easier to understand exactly what the scripts are doing, but this isn't consistent throughout the book and I am not sure what the editorial decision was in not including these types of examples for those scripts that don't have them. My personal opinion is that there should be input and output examples for every recipe in the Bash Cookbook. I liken it to one of my favorite cooking guides, <a href="http://www.amazon.com/exec/obidos/ASIN/B000069YW9/babytoolkit-20">Cooks Illustrated</a>, whose pictures often clear any confusion about preparations for recipes that the text of the recipe may have missed. I think the same holds for sample input and outputs for the tech recipes of the Bash Cookbook. Every recipe, in my opinion should have these examples even if they are only available from <a href="http://www.orielly.com/">O'Reilly's website</a>.<br /><span style="font-weight: bold;font-size:130%;" ><br />Conclusion<br /></span><br />Nitpicks and suggestions aside, this is a great bash scripting resource and should find a good home on any scripter's bookshelf. It provides enough instruction to help a new-ish user understand the deeper power of bash scripting while having enough breadth and depth to serve as an invaluable resource for the experienced scripting guru.<br /><span style="font-weight: bold;font-size:130%;" ><br />Book Information<br /></span><br /><span style="font-weight: bold;"><br />Title:</span> <a href="http://www.amazon.com/exec/obidos/ASIN/0596526784/babytoolkit-20">Bash Cookbook</a><br /><span style="font-weight: bold;">Authors:</span> Carl Albing, JP Vossen & Cameron Newham<br /><span style="font-weight: bold;">Paperback:</span> 622 pages<br /><span style="font-weight: bold;">Publisher:</span> O'Reilly Media, Inc., 1 edition (May 24, 2007)<br /><span style="font-weight: bold;">ISBN:</span> 0596526784Jimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com2tag:blogger.com,1999:blog-8849323713787808858.post-38896022678468714622008-05-20T08:25:00.000-07:002008-05-20T08:47:30.997-07:00Python Tip: Checking to see if your Python installation has support for SSLI was trying to figure out if my installation of Python was compiled with SSL support and found it to be non-intuitive if you didn't compile Python for yourself.<br /><br />So, to check if you have SSL support configured with your installation of Python, go to your command prompt and type:<br /><br /><pre><br />python<br /></pre><br /><br />and you'll get the Python interactive shell (that will look something like this):<br /><br /><pre><br />Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42) <br />[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2<br />Type "help", "copyright", "credits" or "license" for more information.<br />>>><br /></pre><br /><br />At the <i>>>></i> prompt, type <i>import socket</i>:<br /><br /><pre><br />Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42) <br />[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2<br />Type "help", "copyright", "credits" or "license" for more information.<br />>>> import socket<br /></pre><br /><br />Then check for the ssl attribute, by typing <i>hasattr(socket, "ssl")</i> at the <i>>>></i> prompt and look for a <i>True</i> or <i>False</i> response:<br /><br /><pre><br />Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42) <br />[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2<br />Type "help", "copyright", "credits" or "license" for more information.<br />>>> import socket<br />>>> hasattr(socket, "ssl")<br />True<br />>>><br /></pre><br /><br />A <i>True</i> response means that SSL is compiled in your Python installation.<br /><br />Good luck.<br /><br />If you want some good books on learning to use Python, I highly recommend <a href="http://www.amazon.com/exec/obidos/ASIN/159059519X/babytoolkit-20">Beginning Python: From Novice to Professional by Magnus Hetland</a> and <a href="http://www.amazon.com/exec/obidos/ASIN/0596513984/babytoolkit-20">Learning Python by Mark Lutz and David Ascher</a>. I am currently using both books to get up to speed on the Python language and I am really enjoying working with both of them.Jimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com0tag:blogger.com,1999:blog-8849323713787808858.post-61694828959915101322008-05-07T13:39:00.000-07:002008-08-29T09:59:52.326-07:00Bash Tip: Proving a Negative with grep and diffI stumbled across an interesting problem a while back. Given a set of data how do you determine correlating data that isn't there?<br /><br />I was given a file containing a list of names and some other lines that indicated a successful condition. If a name on one line was followed by a success statement, then the condition was successful for the previous line. However, if a name was followed by another name, then the condition had failed for the first name.<br /><br />Confused already? Let's look at an example and see if that clears things up. Say we have a file, tally.txt, whose contents look something like this:<br /><pre><br />Doug<br />Jim<br />voteSuccess<br />Diana<br />voteSuccess<br />Thomas<br />voteSuccess<br />Drew<br />Elizabeth<br />Chris<br />Adrienne<br />voteSuccess<br />Nicholas<br />voteSuccess<br />Anita<br />Greg<br />Jacob<br />Trudy<br />voteSuccess<br />Alex<br />voteSuccess<br />Richard<br />voteSuccess<br />Donald<br />Sam<br />Steve<br />Bob<br />Nathan<br />voteSuccess<br />Penelope<br />voteSuccess<br />Bishop<br />voteSuccess<br />Dustin<br />voteSuccess<br />Ron<br />George<br />Henry<br />voteSuccess<br />Arthur<br />Reggie<br />voteSuccess<br /></pre><br /><br />Here is the same file with line numbers added to make things clearer:<br /><br /><pre><br />1 Doug<br />2 Jim<br />3 voteSuccess<br />4 Diana<br />5 voteSuccess<br />6 Thomas<br />7 voteSuccess<br />8 Drew<br />9 Elizabeth<br />10 Chris<br />11 Adrienne<br />12 voteSuccess<br />13 Nicholas<br />14 voteSuccess<br />15 Anita<br />16 Greg<br />17 Jacob<br />18 Trudy<br />19 voteSuccess<br />20 Alex<br />21 voteSuccess<br />22 Richard<br />23 voteSuccess<br />24 Donald<br />25 Sam<br />26 Steve<br />27 Bob<br />28 Nathan<br />29 voteSuccess<br />30 Penelope<br />31 voteSuccess<br />32 Bishop<br />33 voteSuccess<br />34 Dustin<br />35 voteSuccess<br />36 Ron<br />37 George<br />38 Henry<br />39 voteSuccess<br />40 Arthur<br />41 Reggie<br />42 voteSuccess<br /></pre><br /><br />Lines 3, 5, 7, 12, 14, 19, 21, 23, 29, 31, 33, 35, 39, and 42 all indicate a success condition (voteSuccess). By the conventions of the file, that means that the people on the preceding lines actually had the success (lines 2, 4, 6, 11, 13, 18, 20, 22, 28, 30, 32, 34, 38, and respectively). The problem is that we want to find out who wasn't able to successfully vote. We need to find some way to extract those that voted successfully from the file and only leave those that weren't able to vote.<br /><br />It should be noted that I simplified this example quite a bit. The success condition string (voteSuccess) could actually be one of a host of things, so it is not just one known string that we can work against, but it is good enough for this exercise.<br /><br />Of course this whole situation would be a lot easier if the program that created this file placed some sort of indication of failure after the names of the people that didn't have success in voting. Unfortunately, in many instances, we're often stuck with the formats we're given and have to find a way to make them work.<br /><br />After a little thought, I came up with a psuedo algorithm that I thought might solve the problem:<br /><ol><br /><li>Correlate all the success conditions with the appropriate people.</li><br /><li>Strip these into a file of successful voters sorted alphabetically.</li><br /><li>Strip out all the status messages, leaving only voters, and sort them alphabetically into another file of all voters.</li><br /><li>Check the difference between the files. As the successful voters will be in both files, only those that failed will be different.</li><br /></ol><br />For step one, we'll use a somewhat current version of the grep utility (the one I used was 2.5.1, you can find the version by typing <i>grep -V</i>) to print all the lines of tally.txt that contain <i>voteSuccess</i> and every line above them. The <i>-B</i> switch for <i>grep</i> prints however many lines you want above the string that you were looking for. Typing:<br /><pre><br />grep -B 1 voteSuccess tally.txt<br /></pre><br />You'll notice that the <i>-B</i> switch prints <i>--</i> between contiguous blocks of matches.<br /><pre><br />Jim<br />voteSuccess<br />Diana<br />voteSuccess<br />Thomas<br />voteSuccess<br />--<br />Adrienne<br />voteSuccess<br />Nicholas<br />voteSuccess<br />--<br />Trudy<br />voteSuccess<br />Alex<br />voteSuccess<br />Richard<br />voteSuccess<br />--<br />Nathan<br />voteSuccess<br />Penelope<br />voteSuccess<br />Bishop<br />voteSuccess<br />Dustin<br />voteSuccess<br />--<br />Henry<br />voteSuccess<br />--<br />Reggie<br />voteSuccess<br /></pre><br />We'll clean those out by piping the output into an invert match of grep.<br /><pre><br />grep -B 1 voteSuccess tally.txt | grep -v ^[--]<br /></pre><br />Now we'll clean out the voteSuccess condition statements and sort the output.<br /><pre><br />grep -B 1 voteSuccess tally.txt | grep -v ^[--] | grep -v voteSuccess | sort<br /></pre><br />Our output from the first command sequence looks like this:<br /><pre><br />Adrienne<br />Alex<br />Bishop<br />Diana<br />Dustin<br />Henry<br />Jim<br />Nathan<br />Nicholas<br />Penelope<br />Reggie<br />Richard<br />Thomas<br />Trudy<br /></pre><br />Now that we have a list of the successful voters, let's redirect it to the file, successfulvoters.txt, that we'll later use to ferret out the failed voters.<br /><pre><br />grep -B 1 voteSuccess tally.txt | grep -v ^[--] | grep -v voteSuccess | sort > successfulvoters.txt<br /></pre><br />Next, we need to pull together a sorted list of all voters. This is pretty easy, all we have to do is an inverted search for the term <i>voteSuccess</i>. The only things left will be the names of all the voters which we can sort and redirect into the file allvoters.txt<br /><pre><br />grep -v voteSuccess tally.txt | sort > allvoters.txt<br /></pre><br />Finally, we'll compares the successfulvoters.txt and allvoters.txt files using the <i>diff</i> utility. As <i>diff</i> can be verbose, we'll ask it to output an <i>ed</i> (line editor) script by employing the <i>-e</i> switch. These script instructions will highlight what needs to happen to the successfulvoters.txt file in order to make it look like the allvoters.txt file... which is add back all the failed users.<br /><br />The failed users that we were trying to figure out how to isolate.<br /><br />Let's compare files:<br /><pre><br />diff -e successfulvoters.txt allvoters.txt<br /></pre><br />That gives us the following:<br /><pre><br />12a<br />Ron<br />Sam<br />Steve<br />.<br />6a<br />Jacob<br />.<br />5a<br />Elizabeth<br />George<br />Greg<br />.<br />4a<br />Donald<br />Doug<br />Drew<br />.<br />3a<br />Bob<br />Chris<br />.<br />2a<br />Anita<br />Arthur<br />.<br /></pre><br /><br />If you look at this output upside down, you can follow along in the successfullvoters.txt file and see where these additions would be added in order to make a complete list of users. If you can't do it mentally, I've flipped the output here:<br /><pre><br />.<br />Arthur<br />Anita<br />2a<br />.<br />Chris<br />Bob<br />3a<br />.<br />Drew<br />Doug<br />Donald<br />4a<br />.<br />Greg<br />George<br />Elizabeth<br />5a<br />.<br />Jacob<br />6a<br />.<br />Steve<br />Sam<br />Ron<br />12a<br /></pre><br />However, we just need the usernames and not the commands for the <i>ed</i> utility. If we get rid of every line that starts with a number (our voter names don't start with numbers) and every line with a period (.), and then run that through <i>sort</i>, we should have an alphabetical list of people that couldn't successfully vote.<br /><br />To get rid of any line that begins for a number, we'll do an invert search on the output of the <i>diff</i> command. The expression <i>^[[:digit:]]</i> uses the carat (^) character to denote starting the line and the shorthand expression <i>[[:digit:]]</i> to denote any number. That will just leave us to content with the periods, which we can remove by piping this output into yet another inverted <i>grep</i> search asking to return any line that doesn't contain them, <i>[.]</i>. Then we sort the output to make it more useable.<br /><pre><br />diff -e successfulvoters.txt allvoters.txt | grep -v ^[[:digit:]] | grep -v [.] | sort<br /></pre><br />And that's that!<br /><br />We can put it all together in a quick and dirty bash script that will parse out the failed voters given a file name to process.<br /><br /><pre><br />#!/bin/bash<br /><br /># Take the filename from the command line and stuff it into a variable<br />TALLY=$1<br /><br /># Find the successful voters<br />grep -B 1 voteSuccess $TALLY | grep -v ^[--] | grep -v voteSuccess | sort > successfulvoters.txt<br /><br /># Find all the voters<br />grep -v voteSuccess $TALLY | sort > allvoters.txt<br /><br /># Find the difference between the successful voters<br /># and all the possible voters (ie the failed voters)<br />diff -e successfulvoters.txt allvoters.txt | grep -v ^[[:digit:]] | grep -v [.] | sort<br /></pre><br />Does anyone have other ideas how to tackle this problem? While the test case was relatively small, the actual data set contained tens of thousands of entries.<br /><br />I see a lot of redundancy in this solution with its two passes over the tally file. On the other hand, I had a working solution in about 15 minutes.<br /><br />I have tested this a couple of times and it looks to work on all my data sets. Perhaps there's a problem that I am not seeing and, if so, please speak up and let me know.<br /><br />How would you solve this problem?<br /><br />Do you have any file processing war stories? Tricks of the trade you'd like to share?<br /><br />Also, I have a couple of texts that I have use to flesh out my understanding of scripting and the bash shell. The first is the O'Reilly book <a href="http://www.amazon.com/exec/obidos/ASIN/0596009658/babytoolkit-20">Learning the bash Shell by Cameron Newham</a>. It is a step by step introduction to bash shell concepts and includes a good overview of many standard shell tools and techniques. <br /><br />I also really like the more general book by Stephen Kochan and Patrick Wood titled <a href="http://www.amazon.com/exec/obidos/ASIN/0672324903/babytoolkit-20">Unix Shell Programming</a> (3rd ed). Kochan and Wood write the book to the <a href="http://en.wikipedia.org/wiki/POSIX">POSIX standard</a> for shells which should help in writing maintainable and portable scripts, however they also make an effort to point out how each shell differs in its approach. It has its faults, as most books do, but it is solid nonetheless.Jimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com0tag:blogger.com,1999:blog-8849323713787808858.post-81480527564698452682008-05-07T13:36:00.000-07:002008-09-24T06:25:55.238-07:00Bash Tip: Finding a Line and the One Following itI was recently asked the following question:<br /><blockquote>I need to identify lines containing a string in a file and extract that line and<br />the next line from the file. There might be mutiple occurrences in the file.<br /><br />In this example file I need to scan for "gottoget" and then extract line 3<br />and 4 as well as lines 6 and 7<br /><br />Example file:<br />line 1<br />line 2<br />gottoget line 3<br />want this line as well line 4<br />line 5<br />gottoget line 6<br />ok this one must come with line 7<br />line 8<br />line 9<br />line 10<br /><br />Hope you can help.</blockquote><br /><br />I think this puzzle is easily solved through the use of <span style="font-style: italic;">grep</span>'s <span style="font-style: italic;">-A</span> flag.<br /><br />According to the <span style="font-style: italic;">man</span> page for grep (<span style="font-style: italic;">man grep</span>), the <span style="font-style: italic;">-A</span> flag prints the number of lines specified after the matching lines. It sounds like <span style="font-style: italic;">grep -A 1 gottotext examplefile</span> should do the trick. This line will grab the line containing the string we're looking for ("gottotext" in your example) and the first line after that matching line. If we set <span style="font-style: italic;">grep</span> up this way, we get the following:<br /><br /><pre><br />gottoget line 3<br />want this line as well line 4<br />--<br />gottoget line 6<br />ok this one must come with line 7<br /></pre><br /><br />The <span style="font-style: italic;">--</span> line separates contiguous matches. If you don't want that, the lines are easily removed with another <span style="font-style: italic;">grep</span> filter ( <span style="font-style: italic;">| grep -v ^[--]</span>) which says to show everything but lines that begin wi<span style="font-style: italic;"></span>th the <span style="font-style: italic;">--</span> characters. If you have <span style="font-style: italic;">-- </span>characters that are legitimate at the beginning of some lines in your data, you may need to play around a bit to only filter out these unnecessary ones.<br /><br />Putting it all together, we get:<br /><br /><pre><br />grep -A 1 gottoget examplefile | grep -v ^[--]<br /></pre><br /><br />Giving us the cleaned up output of:<br /><br /><pre><br />gottoget line 3<br />want this line as well line 4<br />gottoget line 6<br />ok this one must come with line 7<br /></pre><br /><br />And that's it. A simple application of some grep statements provides the answer.<br /><br />I highly recommend <a href="http://www.amazon.com/exec/obidos/ASIN/0672324903/babytoolkit-20">Unix Shell Programming (3rd edition)</a> by Stephen Kochan and Patrick Wood if you are interested improving your understanding of shell scripting. Kochan and Wood do a very thorough job (using plenty of code examples) exploring various aspects of essential shell scripting tools and techniques.<br /><br />Post a comment if you have a different or better way of handling this puzzle.<br /><br />Take care.Jimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com0tag:blogger.com,1999:blog-8849323713787808858.post-35610965650932198442008-04-29T11:57:00.000-07:002008-09-24T06:30:36.983-07:00Bash Tip: Reverse Sorting Lists in the ShellEvery once in a while I check my site logs and find a common search phrase in referrals from search engines. Often the visitors appear to leave immediately; presumably because the page they landed on didn't answer their question.<br /><br />A phrase that has been appearing quite frequently lately is "<span style="font-style: italic;">bash reverse sort list</span>".<br /><br />I can't tell exactly what they mean by their search query, so we'll take a couple of cracks at it.<br /><br />My first thought is that they might be looking to reverse the output of the command line tool <span style="font-style: italic;">ls</span>.<br /><br />Say we have a directory and we see the following files when we run <span style="font-style: italic;">ls</span>:<br /><br /><pre><br />a aa ab b bd be c cqw cw d de defg h hi hij hijk i ii iii ij<br /></pre><br /><br />To get a simple reverse listing of those files, we should use the <span style="font-style: italic;">-r</span> switch for <span style="font-style: italic;">ls</span>. typing <span style="font-style: italic;">ls -r</span> in the same directory yields:<br /><br /><pre><br />ij iii ii i hijk hij hi h defg de d cw cqw c be bd b ab aa a<br /></pre><br /><br />Having your file list reversed in a horizontal line isn't always useful when you are looking for a vertical list. It just takes a little bit of extra work to turn your list on its side if that's what you need.<br /><br />First, we'll use the <span style="font-style: italic;">-l</span> switch of <span style="font-style: italic;">ls</span> to show the long listing for the files. Typing <span style="font-style: italic;">ls -lr</span> gives us a reverse listing of our files in a vertical format.<br /><br /><pre><br />total 0<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 ij<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 iii<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 ii<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 i<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 hijk<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 hij<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 hi<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 h<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 defg<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 de<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 d<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 cw<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 cqw<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 c<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 be<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 bd<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 b<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 ab<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 aa<br />-rw-r--r-- 1 jjones jjones 0 2007-05-07 21:23 a<br /></pre><br /><br />That's a bit more verbose than what I expect we're looking for, so we'll need to employ a couple of more tools to trim away the fat.<br /><br />All we want is the last column (8, if you consider the columns delimited by spaces) of information. This is where the <span style="font-style: italic;">cut</span> command comes in handy. It does exactly what the name implies by slicing and dicing data in multiple handy ways.<br /><br />By default, the <span style="font-style: italic;">cut</span> command treats data as fields separated by tabs. By sending the output of our <span style="font-style: italic;">ls -lr</span> command as input to the <span style="font-style: italic;">cut</span> command while changing the default delimiter character with the <span style="font-style: italic;">-d</span> switch, we can filter out all but the 8th column. So far our command looks like this, <span style="font-style: italic;">ls -lr | cut -d" " -f8</span>, and our ouput looks like this:<br /><br /><pre><br /><br />ij<br />iii<br />ii<br />i<br />hijk<br />hij<br />hi<br />h<br />defg<br />de<br />d<br />cw<br />cqw<br />c<br />be<br />bd<br />b<br />ab<br />aa<br />a<br /></pre><br /><br />Almost perfect. However, you'll notice one small problem at the top of the list. There's an extra blank. If you look at the original output of <span style="font-style: italic;">ls -lr</span>, it's quickly becomes clear where the blank line came from. The <span style="font-style: italic;">total 0</span> line in the original output had only two fields, <span style="font-style: italic;">total</span> and <span style="font-style: italic;">0</span>, leaving nothing but a blank when cut went looking for the eighth field.<br /><br />It's not too difficult a job to clean this up with a little creative application of the <span style="font-style: italic;">grep</span> command. We'll use the <span style="font-style: italic;">-v</span> or inverse match switch of <span style="font-style: italic;">grep</span> (otherwise known as "show me everything but") of a line with only a beginning, represented by the carat (^) symbol, and an end, represented by the dollar sign ($) and nothing in between, or <span style="font-style: italic;">-v ^$</span>.<br /><br />Putting it all together as ls -lr | cut -d" " -f8 | grep -v ^$ successfully removes the blank line from our vertical reverse sorted list of files.<br /><br /><pre><br />ij<br />iii<br />ii<br />i<br />hijk<br />hij<br />hi<br />h<br />defg<br />de<br />d<br />cw<br />cqw<br />c<br />be<br />bd<br />b<br />ab<br />aa<br />a<br /></pre><br /><br />Another list you might like to sort is one contained in a file. <span style="font-style: italic;">ls</span> isn't going to help us with this one, but the <span style="font-style: italic;">sort</span> command is here to help.<br /><br />By default, <span style="font-style: italic;">sort</span> will sort a list in a file by the first field as delimited by white and non-white space. Taking an example file (sort.txt) containing the following:<br /><br /><pre><br />a<br />b<br />bd<br />hij<br />be<br />aa<br />cqw<br />ab<br />c<br />cw<br />d<br />de<br />iii<br />defg<br />h<br />hi<br />hijk<br />i<br />ii<br />ij<br /></pre><br /><br />So, running <span style="font-style: italic;">sort</span> against sort.txt results in:<br /><br /><pre><br />a<br />aa<br />ab<br />b<br />bd<br />be<br />c<br />cqw<br />cw<br />d<br />de<br />defg<br />h<br />hi<br />hij<br />hijk<br />i<br />ii<br />iii<br />ij<br /></pre><br /><br />The <span style="font-style: italic;">sort</span> command also offers a reverse sort option through the <span style="font-style: italic;">-r</span> switch. Running <span style="font-style: italic;">sort -r</span> against sort.txt (<span style="font-style: italic;">sort -r sort.txt</span>) results in:<br /><br /><pre><br />ij<br />iii<br />ii<br />i<br />hijk<br />hij<br />hi<br />h<br />defg<br />de<br />d<br />cw<br />cqw<br />c<br />be<br />bd<br />b<br />ab<br />aa<br />a<br /></pre><br /><br />I hope this answers some of the basic questions about reverse sorting lists. For more information check out the manual pages for <span style="font-style: italic;">ls</span> and <span style="font-style: italic;">sort</span> (<span style="font-style: italic;">man ls</span> and <span style="font-style: italic;">man sort</span>).<br /><br />However, you might just want your list flipped on its head, with no sorting whatsoever. Say you have the list:<br /><br /><pre><br />a<br />d<br />c<br />b<br /></pre><br /><br />You want it like to look like this:<br /><br /><pre><br />b<br />c<br />d<br />a<br /></pre><br /><br />As it turns out, there is a command just for that purpose called <span style="font-style: italic;">tac</span>. Where <span style="font-style: italic;">cat</span> will concatenate the contents of a file to the screen (standard output), <span style="font-style: italic;">tac</span> will do the same after reversing the contents of a file.<br /><br />Take the text of the 1st Amendment to the US Constitution, for example.<br /><br /><pre><br />Congress shall make no law respecting an establishment of religion,<br />or prohibiting the free exercise thereof;<br />or abridging the freedom of speech,<br />or of the press;<br />or the right of the people peaceably to assemble,<br />and to petition the Government for a redress of grievances.<br /></pre><br /><br />Running <span style="font-style: italic;">tac</span> against these lines compeletely reverses them:<br /><br /><pre><br />and to petition the Government for a redress of grievances.<br />or the right of the people peaceably to assemble,<br />or of the press;<br />or abridging the freedom of speech,<br />or prohibiting the free exercise thereof;<br />Congress shall make no law respecting an establishment of religion,<br /></pre><br /><br />Whereas if we had used <span style="font-style: italic;">sort</span>, the output would look slightly different:<br /><br /><pre><br />and to petition the Government for a redress of grievances.<br />Congress shall make no law respecting an establishment of religion,<br />or abridging the freedom of speech,<br />or of the press;<br />or prohibiting the free exercise thereof;<br />or the right of the people peaceably to assemble,<br /></pre><br /><br /><br />If your list isn't vertical with items separated by a newline, you can use <span style="font-style: italic;">tac</span>'s <span style="font-style: italic;">-s</span> switch, similar to <span style="font-style: italic;">cut</span>'s <span style="font-style: italic;">-d</span> switch, to identify a different separator.<br /><br /><span style="font-weight: bold;">Update 1: </span>A helpful reader pointed out that the <span style="font-style: italic;">ls</span> examples could be a lot smaller with the application of the <span style="font-style: italic;">-1</span> switch to the <span style="font-style: italic;">ls</span> command. This switch tells the standard <span style="font-style: italic;">ls</span> command to print one file per line. When combined with the reverse, <span style="font-style: italic;">-r</span>, switch, we get a reverse list of files in a vertical as opposed to the standard horizontal layout.<br /><br />In the end, just typing<br /><br /><pre><br />ls -1r<br /></pre><br /><br />will result in this list of files<br /><br /><pre><br />a aa ab b bd be c cqw cw d de defg h hi hij hijk i ii iii ij<br /></pre><br /><br />being printed like this<br /><br /><pre><br />ij<br />iii<br />ii<br />i<br />hijk<br />hij<br />hi<br />h<br />defg<br />de<br />d<br />cw<br />cqw<br />c<br />be<br />bd<br />b<br />ab<br />aa<br />a<br /></pre><br /><br /><span style="font-weight: bold;">Update 2:</span> It turns out that I didn't cover how to reverse sort a horizontal line. Since it is a little long, you can check out my solution in this post, <a href="http://morlockhq.blogspot.com/2008/09/bash-tip-reverse-sorting-lists-revisted.html">Bash Tip: Reverse Sorting Lists Revisted; Reversing a Horizontal List</a>.<br /><br />--<br /><br />I hope these tips help everyone out. If you want more resources on shell scripting, I highly recommend <a href="http://www.amazon.com/exec/obidos/ASIN/0672324903/babytoolkit-20">Unix Shell Programming (3rd edition)</a> by Stephen Kochan and Patrick Wood if you are interested improving your understanding of shell scripting. Kochan and Wood do a very thorough job (using plenty of code examples) exploring various aspects of essential shell scripting tools and techniquesJimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com2tag:blogger.com,1999:blog-8849323713787808858.post-51249557320566827822008-04-29T11:39:00.000-07:002008-04-29T11:41:19.538-07:00Bash Tip: Extracting a Range of Lines with sedA little puzzle came across my desk the other day. I was asked how we could pull a range of lines from a file.<br /><br />Say the file has ten lines:<br /><br /><pre><br />Line 1<br />Line 2<br />Line 3<br />Line 4<br />Line 5<br />Line 6<br />Line 7<br />Line 8<br />Line 9<br />Line 10<br /></pre><br /><br />We want to extract only lines five through eight from the file.<br /><br />This isn't a hard problem, and there are probably a thousand ways to do this. However, I chose to use a simple <i>sed</i> command.<br /><br />By typing the following we can get the result we want.<br /><br /><pre><br />sed -n '5,8p;8q' filename<br /></pre><br /><br />This line basically tells <i>sed</i> to start at line 5 and print until line 8 and then end processing at line 8. This simple construct works equally well with very large files and wider ranges of lines. Your output will look something like this:<br /><br /><pre><br />Line 5<br />Line 6<br />Line 7<br />Line 8<br /></pre><br /><br />It also works for extracting single lines if you change the line to look something like this:<br /><br /><pre><br />sed -n '5p;5q' filename<br /></pre><br /><br />The above line prints the single line (in this case "Line 5") and then stops processing the file.<br /><br />I recommend the books <a href="http://www.amazon.com/exec/obidos/ASIN/0596005954/babytoolkit-20">Classic Shell Scripting</a> by Arnold Robbins and Nelson H.F. Beebe and O'Reilly's <a href="http://www.amazon.com/exec/obidos/ASIN/1565922255/babytoolkit-20">Sed and Awk (second edition)</a> by Dale Dougherty and Arnold Robbins if you would like additional resources for learning how to effectively leverage the <i>sed</i> utility.<br /><br />How do you extract lines from a file? What tools and techniques work for you?Jimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com3tag:blogger.com,1999:blog-8849323713787808858.post-73571125654643947452008-01-28T12:45:00.000-08:002008-05-20T08:48:16.475-07:00Komodo Tip: Launching a Python Shell and Running Code from Komodo Edit 4.2 on Linux<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjY7dZ62mQ4NUQhDDJEG864H5OhXUq_Zvgk-JPbxwb51PZB51HbusVwQONQBPPm8f9DgZcdXBaEutSfcvkyTdAaH00OHB8gISWBjdEuTsQ9HwdI_L3ZasFfLtOekYekUrW4x2nUo-dHmwQ/s1600-h/komodo.jpg"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjY7dZ62mQ4NUQhDDJEG864H5OhXUq_Zvgk-JPbxwb51PZB51HbusVwQONQBPPm8f9DgZcdXBaEutSfcvkyTdAaH00OHB8gISWBjdEuTsQ9HwdI_L3ZasFfLtOekYekUrW4x2nUo-dHmwQ/s320/komodo.jpg" alt="" id="BLOGGER_PHOTO_ID_5160632270738980002" border="0" /></a>I have been teaching myself <a href="http://www.python.org/">python</a> lately and as part of that I have been trying out a few editors including <a href="http://www.vim.org/">vim</a>, <a href="http://geany.uvena.de/">geany</a>, and (most recently) <a href="http://www.activestate.com/Products/komodo_edit/">Komodo Edit 4.2</a> from the folks at <a href="http://www.activestate.com/">Active State</a> (makers of fine cross platform scripting tools and language ports).<br /><br />Komodo Edit appears to be a stripped down version of Active States more full featured<a href="http://www.activestate.com/Products/komodo_ide/"> Komodo IDE</a>. Even though it is stripped down, it still has a <a href="http://www.activestate.com/Products/komodo_edit/">some nice features</a>.<br /><br />While the Komodo Edit doesn't have all the built in features of the full-fledged IDE, the inclusion of the extensible <span style="font-weight: bold;">Toolbox</span> allows you to approximate some tools and integration that turns out to be quite handy.<br /><br /><span style="font-weight: bold;font-size:130%;" >Launching a Python Shell</span><br /><br />The Komodo IDE has a nice integrated shell for python that makes trying out code quite easy. This is one of the features that was left out of Komodo Edit, but we can make launching a shell a little easier by creating a <span style="font-weight: bold;">Toolbox</span> entry that you can then bind to a key combination and launch from within the editor.<br /><br />To get the entry:<br /><ol><li>Launch <span style="font-weight: bold;">Komodo Edit</span></li><li>Click the <span style="font-weight: bold;">Toolbox</span> drop down menu</li><li>Choose the <span style="font-weight: bold;">Add</span> menu item</li><li>Then, click the <span style="font-weight: bold;">New Command</span> item</li><li>In the <span style="font-weight: bold;">Add Command</span> window, type<span style="font-style: italic;"> Python Shell</span> in the first feild</li><li>Then type, <span style="font-style: italic;">gnome-terminal -x python</span> (or your terminal emulator of choice with the appropriate execute switch)</li><li>The choose, <span style="font-weight: bold;">No Console (GUI Application)</span> from the <span style="font-weight: bold;">Run In:</span> field</li><li>Next, click the <span style="font-weight: bold;">Key Binding</span> tab at the top and assign a keyboard combo that you can use to easily launch a python shell.</li></ol><br /><br /><span style="font-weight: bold;font-size:130%;" >Running a Python File</span><br /><br />Similarly you can create a <span style="font-weight: bold;">Run Command</span> to use python to execute the current file that you are working with.<br /><br />To get the entry:<br /><ol><li>Launch <span style="font-weight: bold;">Komodo Edit</span></li><li>Click the <span style="font-weight: bold;">Toolbox</span> drop down menu</li><li>Choose the <span style="font-weight: bold;">Add</span> menu item</li><li>Then, click the <span style="font-weight: bold;">New Command</span> item</li><li>In the <span style="font-weight: bold;">Add Command</span> window, type<span style="font-style: italic;"> Run Python File</span> in the first feild</li><li>Then type, <span style="font-style: italic;">%(python) %F</span></li><li>The choose, <span style="font-weight: bold;">Command Output Tab</span> from the <span style="font-weight: bold;">Run In:</span> field</li><li>Next, click the <span style="font-weight: bold;">Key Binding</span> tab at the top and assign a keyboard combo that you can use to easily launch a python shell.</li></ol>Make sure to save the file before launching the <span style="font-weight: bold;">Run Python File</span> tool. If you are successful, your code will appear in a window at the bottom of the editor. If you get a stack trace, you can double click on the error and your cursor will jump back to the editor window to the line that is causing the problem.<br /><br />I will post any more tricks in Komodo Edit as I come across them. If you have any tips or tricks, please post them to the comments.<br /><br /><br /><span style="font-size:78%;">CC photo credit: <a href="http://www.flickr.com/people/fhsu/"> Fred Hsu</a></span>Jimhttp://www.blogger.com/profile/05953752580633741241noreply@blogger.com1