RubyGuides
Share this post!

All posts by Jesus Castello

The tree command

With the tree command in Linux you can get a tree representation of a directory structure. Without any arguments it will start of the current dir and recursively go into each subdir to show a complete hierarchy.

tree

.
├── 1
│   ├── 44
│   ├── aa
│   ├── bb
│   └── ff
└── 2
├── cc
└── dd

3 directories, 5 files

This is just some dirs and files I made for testing, but if you run this on a real directory you will get a lot of output, to avoid this you can use the -L option to limit the depth.

tree -L 1

.
├── 1
└── 2

Well that’s a bit better, you can also get other useful information like permissions using the -p option:

# tree -p
.
├── [drw-r-----] 1
│   ├── [drwxr-xr-x] 44
│   ├── [-rw-r--r--] aa
│   ├── [-rw-r--r--] bb
│   └── [-rw-r--r--] ff
└── [drwxr-xr-x] 2
├── [-rw-r--r--] cc
└── [-rw-r--r--] dd

Another useful one is -u to show the owners of the files:

# tree -u
.
├── [root ] 1
│   ├── [root ] 44
│   ├── [matu ] aa
│   ├── [matu ] bb
│   └── [matu ] ff
└── [root ] 2
├── [root ] cc
└── [root ] dd

Other options that can come in handy are -d to show only dirs, and -s to show the size of files, but I will leave these to try on your own.

Intro to Awk

Awk is the ideal tool for most of your output processing/formatting needs. We can use it to carefully select and reformat data fields from stdin or a file and even do stuff like using conditions for what the value must be to print it or not. In fact awk is a programming language in itself, but don’t worry too much about that.

We are going to see an example of how we can print the first and second field of a comma separated list. To start with we will need to tell awk how it should split the fields, in this case by a comma. The option you have to use is -F “field_separator”, so in this case it would be -F ,

I’m going to use a output file from a metasploit module as an example.

We use the print statement which executes once per line of input, notice how it goes between brackets and single quotes or it won’t work, then we specify the field number using a dollar sign, so for field 4 the field variable is $4. Finally, if we want to have our own text or even a tab or newline we need to enclose it between double quotes.

Using conditions

As a second example let’s split /etc/passwd and print the whole line for those users with a uid greater than 1000, for this example we will need to use an if statement and split on colon.

In this example we print the whole line using $0, only for the lines where the third column (the user id) is greater than 1000.

More awk examples

Print three aligned columns: uid, user name, shell.

Add line numbers.

Find and print the biggest UID.

There is a lot more you can do with Awk, one good place to look at is over here.

Four ways to extract files from pcaps

It’s time to extract files from pcaps. If you ever played with packet captures you probably thought it would be cool that you could actually get downloaded files so let’s see not only one way to do this, but four!

1. Wireshark: http export

You can find this at File > Export > Objects > Http, you will be presented with a list of files found in all the http requests. The bad thing about this feature is that even with the latest version (1.6.5 at the time of this writing) you still can’t sort by column or apply any filters which makes finding something specific hard.

2. Wireshark: export bytes

To find this you will have to drill down in the packet you want, depending on the protocol.

Right click > Export selected bytes

extract files from pcap

The advantage of doing it this way is that you can actually extract files from other protocols other than http (like ftp or smb) and you can use display filters.

3. Network miner http://sourceforge.net/projects/networkminer/

Network miner is a tool for network analysis but with a focus on forensic analysis. It can load a pcap and extract files and other data, there is both a free and a commercial version available.

networkminer

4. Chaosreader

This tool will analyze and extract session information and files and create an html report you can open in any browser

chaosreader http-data.pcap

It will create a lot of files so you may want to launch it inside an empty dir or make a new one and use the -D option, then you can open index.html

extract data

Conclusion

If the data crossed the network it has to be there somewhere. In this post we have seen a few tools you can use to uncover these files and extract them for your own benefit.

Parsing HTML in Ruby

If you have ever tried to write a scrapping tool you probably had to deal with parsing HTML. This task can be a bit difficult if you don’t have the right tools. Ruby has this wonderful library called Nokogiri, which makes HTML parsing a walk in the park.

Let’s see some examples.

First install the nokogiri gem with:  gem install nokogiri

Extracting the title

Then create the following script, which contains a basic HTML snippet that will be parsed by Nokogiri. The output will be the page title.

Extracting anchor links

So that was pretty easy, wasn’t it?

Well, it’s doesn’t get much harder than that. For example, if we want all the links from a page we need to use the xpath method on the object we get back from Nokogiri, then we can print the individual attributes of the tag or the text inside the tags:

And that’s it, as you may have already guessed the xpath method uses the Xpath query language, for more info on xpath check out this link.

You can also use CSS selectors, which I find a lot easier to work with. You just need to replace the xpath method with the css method.

Example:

Note: The difference between at_css & css is that the first one only returns the first matched element, but the latter returns ALL matched elements.

To find the correct CSS selector can use your browser’s developer tools.

Summary

In this post you learned about Nokogiri, a tool used to parse (make sense of) HTML source code. You also learned how to use to extract data from the HTML, like the page’s title.

For more on Nokogiri read the documentation here: http://www.rubydoc.info/github/sparklemotion/nokogiri

Ruby String Formatting

Let’s talk about how you can format strings in ruby.

Why would you want to format a string? Well, you may want to do things like have a leading zero even if the number is under 10 (example: 01, 02, 03…), or have some console output nicely formatted in columns.

In other languages you can use the printf function to format strings, and if you have ever used C you are probably familiar with that. To use printf you have to define a list of format specifiers and a list of variables or values.

Getting Started with Ruby String Formatting

While sprintf is also available in Ruby, in this post we will use a more idiomatic way (for some reason the community style guide doesn’t seem to agree on this, but I think that’s ok).

Here is an example:

Output => "Processing of the data has finished in 5 seconds"

In this example, %d is the format specifier (here is a list of available specifiers) and time is the variable we want formatted. A %d format will give us whole numbers only.

If we want to display floating point numbers we need to use %f. We can specify the number of decimal places we want like this: %0.2f.

The 2 here indicates that we want to keep only two decimal places.

Here is an example:

Output => The average is 78.54

Remember that the number will be rounded up. For example, if I used 78.549 in the last example, it would have printed 78.55.

Converting and Padding

You can convert a decimal number and print it as hexadecimal. Using the %x format:

Output => 122 in HEX is 7a

To pad a string:

Use this format for padding a number with as many 0’s as you want: %0<number of zeros>d

Output => The number is 0020

You can also use this ruby string format trick to create aligned columns of text. Replace the 0 with a dash to get this effect:

ruby string format

Alternatively, you can use the .ljust and .rjust methods from the String class to do the same.

Example:

Conclusion

As you have seen ruby & rails string formatting is really easy, it all comes down to understanding the different format specifiers available to you.

I hope you enjoyed this fast trip into the world of output formatting! Don’t forget to subscribe to my newsletter so I can send you more great content 🙂

1 23 24 25