tcpdump for capturing packets and headers

Lots of programs and methods are able to connect to the internet and download webpages. Unfortunately, when you try to do this with your own programs, a ton of stupid webpages have checks that will stop you. They only want their "real users" using regular browsers to download them.

So, I used tcpdump to reverse engineer webpages and figure out exactly how to imitate / pretend to be a regular user,

First, open your browser and clear all cookies, cache, etc. etc. Also, it helps to turn off automatic image loading (Edit -> Preferences -> Content , uncheck "Load images automatically" in Firefox) Then, from the command line, type:

sudo tcpdump -s 6000 -i eth1 -A -w outputFile

Now, visit the webpage with your browser, and do whatever you want your program to do (login, load certain pages, etc. etc.)

tcpdump will be listening and capturing exactly how your browser and this webpage interacted.

Ctrl+C the tcpdump when you're done. Then type:

tcpdump -A -r outputFile > outputFile2

Now you can vi outputFile2 and view how your browser and the webpage interacted.

Your program may need to imitate such things as the User-Agent, Referrer, etc. Look out for mysterious redirects used to fool programs, and make sure to capture all cookies set.

Shell script for loops

To do a list of commands over and over for each subdirectory, type:

sh
for i in `ls -d ./*/`
do
cd $i
(list of commands like mv ../a . or perl blog.pl *blog*.html etc.)
cd ..
done
exit

List only directories in Unix

Much easier way to list only the directories:

ls -d ./*/

Parenthesized search and replace in vi

Parenthesized portions of a regular expression match can be referred to later!

ie \(match1\).*\(match2\) / \1 \2/

\1 will be replaced with whatever match1 held, and \2 will be replaced with whatever match2 held.

ie

:%s/.* .* \(.*\)/\1/

will remove the first two columns just like in the below blog

vi search and replace with regular expressions

in vi, to search and replace:

:%s/search_phrase/replace_phrase/

with regular expressions:

.* matches one or more characters

so to get rid of the first two columns, separated by spaces

ie

2395 4586 3496
146434 6 3945

:%s/.* .* //

so it becomes:

3496
3945

or you could also do
:%s/[0-9]* [0-9]* //

Reformatting drives in unix

run:

sudo gparted

(sudo apt-get install gparted if it's not already installed in ubuntu)

FAT32 has been working pretty well for R/W access in both windows and unix. Though it has a 4GB filesize limit, and ~20,000 to 50,000 files per folder limit as well.

Execute shell commands from java

(Probably only works on unix)

To just execute without worrying about returned output:

Runtime.getRuntime().exec(new String[] { "sh", "-c", "mv a b" });

(This example would run "mv a b" in the shell)

To execute and retreive the returned output:

Runtime rt = Runtime.getRuntime();
Process p = rt.exec((new String[] { "sh", "-c", "ls *.tmp" }));
try{
p.waitFor();
InputStream is = p.getInputStream();
java.io.DataInputStream din = new java.io.DataInputStream(is);
} catch(Exception e){}

(din will have the returned output of running "ls *.tmp" in the shell)

What line number you're on in vi

To find out what line you're currently at in a file in vi, type

Ctrl + g

This will also give you some other helpful info, like what % of the way through the file you're at and what column you're on.

vi commands - insert at beginning or end of every line

:%s/^/hello/
inserts "hello" at the beginning of every line

:%s/$/world/
inserts "world" at the end of every line

Listing directory sizes in Unix

* makes it list the files in the current directory.

du -sk * | sort -n