RubyGuides
Share this post!

All posts by Jesus Castello

Linux: Monitoring resources – Disk Space

Continuing with our series on Linux resources we are now going to check our disk space. To get a listing of mounted partitions with their size and used space you can use the df -h command, the -h meaning human-readable size format (MBs/GBs), also we can get rid of the virtual filesystems info using grep.

Linux disk space

Ok, so we may be running out of disk space, how can we check which are the biggest culprits?

Using the du command

You can use du with the -s option for a summary view (instead you would get the size of every file) and if you have no idea where to start looking then you can start with the root dir (it may take some time so it should be better do it at off hours). If you have network mounted devices or some other dir you want to ignore you can use the –exclude option.

If you want to get sorted output from du, sort can do this since coreutils 7.5 using the -h option.

Other Linux disk space tools

There is another tool called ncdu which basically does the same as du but it uses a ncurses based interface and let’s you navigate through the results and sort them, on ubuntu based distros it’s just an apt-get away.

ncdu

If you prefer a GUI application you can try with Baobab, which is gnome based or filelight for the KDE guys.

Linux: Monitoring resources – Memory

I’m starting this series of post about Linux monitoring, so you will learn how to check your system resources (ram, cpu, disk). This post starts talking about memory.

You can check memory information using the free command, passing the -m option you will get the output in MBs, let’s see the output from my backtrack VM and discuss its meaning.

What we get here on the first line is our total memory (minus some that is reserved by the kernel), memory used and free memory. Now don’t freak out when you see this value is very low, it’s completely normal, to understand this let’s move on the second line: buffers/cache.

We can see 2 values here: buffers and cache. Buffers is in the ‘used’ column and it represents the actual memory used by running programs. The cache is mainly filled with cached data from disk, this is done to speed up the access to recently used data since it will be in RAM instead of having to load it from disk. The trick here is that the operating system will let go of the cached data to make room for other needs, so we could say that all the cached memory is available memory (the free column on the second line) in fact the System Monitor application in gnome takes these values directly to show used and available memory.

Linux monitoring: memory

Now it’s time for a little experiment to see cached memory in action, we will use the time command to measure how long it takes for a command to complete, in this case I decided to use ls with recursive option. What we are going to see is the difference between uncached and cached load speed.

root@bt:~# time ls -R /opt > /dev/null

real 0m14.516s
user 0m0.320s
sys 0m0.436s

root@bt:~# time ls -R /opt > /dev/null

real 0m0.506s
user 0m0.344s
sys 0m0.080s

As you can see the results are pretty dramatic.

Finally the last line is our Swap usage, this is the dedicated partition we make at install time to help our RAM when it’s getting low, also memory from applications that haven’t been used in a while or when the system needs more ram (for example, to launch a new application) when this happens it’s said that the data is swapped out to disk, to temporarily free some physical ram for applications that you are actually using, when the data from the Swap is needed it’s just brought back to ram (swapped in)

Hope you enjoyed this, even if you didn’t I would like to get some feedback so I know what can be improved, thanks for reading!

Update: Check out this page for more disk cache examples => http://www.linuxatemyram.com/play.html

The tree command

With the tree command in Linux you can get a tree representation of a directory structure. Without any arguments it will start of the current dir and recursively go into each subdir to show a complete hierarchy.

tree

.
├── 1
│   ├── 44
│   ├── aa
│   ├── bb
│   └── ff
└── 2
├── cc
└── dd

3 directories, 5 files

This is just some dirs and files I made for testing, but if you run this on a real directory you will get a lot of output, to avoid this you can use the -L option to limit the depth.

tree -L 1

.
├── 1
└── 2

Well that’s a bit better, you can also get other useful information like permissions using the -p option:

# tree -p
.
├── [drw-r-----] 1
│   ├── [drwxr-xr-x] 44
│   ├── [-rw-r--r--] aa
│   ├── [-rw-r--r--] bb
│   └── [-rw-r--r--] ff
└── [drwxr-xr-x] 2
├── [-rw-r--r--] cc
└── [-rw-r--r--] dd

Another useful one is -u to show the owners of the files:

# tree -u
.
├── [root ] 1
│   ├── [root ] 44
│   ├── [matu ] aa
│   ├── [matu ] bb
│   └── [matu ] ff
└── [root ] 2
├── [root ] cc
└── [root ] dd

Other options that can come in handy are -d to show only dirs, and -s to show the size of files, but I will leave these to try on your own.

Intro to Awk

Awk is the ideal tool for most of your output processing/formatting needs. We can use it to carefully select and reformat data fields from stdin or a file and even do stuff like using conditions for what the value must be to print it or not. In fact awk is a programming language in itself, but don’t worry too much about that.

We are going to see an example of how we can print the first and second field of a comma separated list. To start with we will need to tell awk how it should split the fields, in this case by a comma. The option you have to use is -F “field_separator”, so in this case it would be -F ,

I’m going to use a output file from a metasploit module as an example.

We use the print statement which executes once per line of input, notice how it goes between brackets and single quotes or it won’t work, then we specify the field number using a dollar sign, so for field 4 the field variable is $4. Finally, if we want to have our own text or even a tab or newline we need to enclose it between double quotes.

Using conditions

As a second example let’s split /etc/passwd and print the whole line for those users with a uid greater than 1000, for this example we will need to use an if statement and split on colon.

In this example we print the whole line using $0, only for the lines where the third column (the user id) is greater than 1000.

More awk examples

Print three aligned columns: uid, user name, shell.

Add line numbers.

Find and print the biggest UID.

There is a lot more you can do with Awk, one good place to look at is over here.

Four ways to extract files from pcaps

It’s time to extract files from pcaps. If you ever played with packet captures you probably thought it would be cool that you could actually get downloaded files so let’s see not only one way to do this, but four!

1. Wireshark: http export

You can find this at File > Export > Objects > Http, you will be presented with a list of files found in all the http requests. The bad thing about this feature is that even with the latest version (1.6.5 at the time of this writing) you still can’t sort by column or apply any filters which makes finding something specific hard.

2. Wireshark: export bytes

To find this you will have to drill down in the packet you want, depending on the protocol.

Right click > Export selected bytes

extract files from pcap

The advantage of doing it this way is that you can actually extract files from other protocols other than http (like ftp or smb) and you can use display filters.

3. Network miner http://sourceforge.net/projects/networkminer/

Network miner is a tool for network analysis but with a focus on forensic analysis. It can load a pcap and extract files and other data, there is both a free and a commercial version available.

networkminer

4. Chaosreader

This tool will analyze and extract session information and files and create an html report you can open in any browser

chaosreader http-data.pcap

It will create a lot of files so you may want to launch it inside an empty dir or make a new one and use the -D option, then you can open index.html

extract data

Conclusion

If the data crossed the network it has to be there somewhere. In this post we have seen a few tools you can use to uncover these files and extract them for your own benefit.