December 17, 2008
MAC OS X
MAC Free Utilities
UNIX History
Apple Developers Tools
Install Apache Tomcat 6 on MAC
Mac OS X (Darwin) Releases
What is a MAC OS?
Creating a login hook
MAC VIM
SSH on MAC
Macintosh, Apache, Mysql and PHP
MACMALL
MAC Pro Manual
MAC Developer Program (Students)
MACOSXHINTS
TV Tuner
Cross Platform Software
Unix Open Source to Darwin
Zatto - TV on Your MAC
iSlimPhoto
December 3, 2008
Linux
Open configuration file
$ sudo vi /etc/network/interfaces
Find and remove
iface eth0 inet dhcp
Append new configuration
iface eth0 inet static
address 192.168.1.100
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.254
Save and close file. Restart the network
$ sudo /etc/init.d/networking restart
Shutdown immediately:Manual configuring a WEP connection
shutdown -h now
Reboot immediately:
shutdown -r now
Shutdown at 8 pm:
shutdown -h 20:00
Shutdown in 10 minutes:
shutdown -h +10
Determine internal devices
$ lspci -v | less
Information concerning network devices
$ lshw -C network
$ sudo ifconfig [interface] down
$ sudo dhclient -r [interface]
$ sudo ifconfig [interface] up
$ sudo iwconfig [interface] essid “ESSID_IN_QUOTES”
$ sudo iwconfig [interface] key HEX_KEY <<<-------- If using ASCII Equivalent, this is s:ASCII_KEY (please make note of the prefix s:)
****Additional Comand that may be needed -- sudo iwconfig [interface] key open <<<----See note below
$ sudo iwconfig [interface] mode Managed
$ sudo dhclient [interface]
October 2, 2008
Perl
> find `perl -e 'print "@INC"'` -name '*.pm' -print
To use the CPAN shell tool:
> sudo perl -MCPAN -e shell
> install Spread***::WriteExcel
With this command, the CPAN shell will automatically download the
module, find & retrieve any dependencies like Parse::RecDescent and
File::Temp, and unpack, configure, make, test, and install for you.
> perl -v
To print the configuration summary
> perl -V
Additional Information
August 22, 2008
Unix: vi commands
a | Insert text after the cursor. |
i | Insert text before the cursor. |
o | Open a new line below the line your cursor is currently on. |
A | Insert text at the end of the line. |
I | Insert text at the beginning of the line. |
O | Open a new line above the line your cursor is currently on. |
x | Delete the character beneath the cursor. |
h | Move your cursor to the left by one character. |
l | Move your cursor to the right by one character. |
j | Move your cursor up by one line. |
k | Move your cursor down by one line. |
$ | Move to the end of the line. |
0 | Move to the beginning of the line. |
w | Move forward one word and place the cursor on the first character of that word. |
b | Move backward one word and place the cursor on the first character of that word. |
G | Go to the end of the file. |
1G | Go to line 1 of the file. |
CTRL+g | Report on the current line number that the cursor if on as well as the number of total lines in the file. |
dd | Delete the current line. |
cw | Change word. That is, change the characters from where your cursor is to the next space or punctuation. |
cl | Change letter. Change the character that your cursor is on. |
yy | Copy the line that the cursor if currently on. |
p | Place the contents of either a deleted line or a copied line on the line immediately following the one your cursor is currently on. |
u | Undo the last change. |
:w filename | Save changes. The filename is optional and is only used to specify the name of the file that you would like to save your work as. |
:wq filename | Save changes and quit. Again, the filename is optional. |
ZZ | Save changes and quit. |
:q! | Quit without saving changes. |
/string | Search for next occurrence of string in the file. |
n | Repeat the last search |
. | Repeat the last insert function. |
Unix Commands and Scripts
Make a test directory.
> mv data data1
The move command copies over the previous file. Data1 will be the new file.
> umount -l /media/usb
This command is used to un-mount busy usb drives.
> cat *.nbe grep results awk -F"" '{print $3}' sort uniq
This unix statement concatenates all of the .nbe files, searches for the word 'results' and print the third column of uniq data in sorted order
The cat command is a standard Unix program used to concatenate and display files. The name is from catenate, a synonym of concatenate.
grep is a command line text search utility originally written for Unix. The program's name derives from the UNIX ed comman, g/re/p which performs a similar operation.
Although grep is not strictly an acronym, the letters are taken from global / regular expression / print, a series of instructions for the ed text editor. The grep command searches files or standard input globally for lines matching a given regular expression, and prints them to the program's standard output.
> cat *.txt grep Interesting awk '{print $4}' sort uniqThis statement is utilized with a nmap statement. Used to search and publish the IP address from the scans tagged as 'Interesting'.
> tar -zxvf test123.tar.gz
z -- unzip
x -- extract the file
v -- verbose
f -- forcefully done
How to unzip a tar file?
August 21, 2008
MySQL - Functions
Uploading data into tables from a file.
>DELETE FROM table
Delete all data from table
>LOAD DATA LOCAL INFILE '/location/file' INTO TABLE table (column1);
Load data into a table from a file into particular columns.
>LOAD DATA LOCAL INFILE '/location/file' INTO TABLE table (column1) SET (column2) = 'data';
Add a value to the column that is not in the source file. Ensure the '( )' are around the column1 name.
>SELECT version();
Determine the version on MySQL you are running.
>SELECT now();
Determine the date and time.
>MYSQLDUMP -u username -p databasename > filename.sql
Backup the entire database from the command prompt.
>DELETE FROM table where Field = 'value';
To delete a row from the database.
>Select * From table WHERE value NOT IN (SELECT value FROM table Union SELECT value FROM table);
Query data from two separate tables where it does not exist in the master table.
>SELECT DISTINCT field FROM table INTO OUTFILE '/location/filename';
Directs the output to a file.
>CREATE TABLE Name Select * from Orginal_Table;
Create a table structure from another table.
>Select a.IP, b.nessus as nessus, c.patchlink as patchlink, d.nmap as nmap from JASON2_Inv as a left join Scan_Nessus_Inv as b on a.IP = b.IP left join Scan_Patchlink_Inv as c on a.IP = c.IP left join Scan_NMAP_Inv as d on a.IP = d.IP;
This is an example of a left join with four tables.
Import Data from a File to MySQL
- Save your Excel data as a csv file (In Excel 2007 using Save As)
- Check the saved file using a text editor such as Notepad to see what it actually looks like, i.e. what delimiter was used etc.
- Start the MySQL Command Prompt (I'm lazy so I usually do this from the MySQL Query Browser - Tools - MySQL Command Line Client to avoid having to enter username and password etc.)
- Enter this command:
LOAD DATA LOCAL INFILE 'C:\\temp\\yourfile.csv' INTO TABLE database.table FIELDS TERMINATED BY ';' ENCLOSED BY '"' LINES TERMINATED BY '\r\n' (field1, field2); - Done!
August 19, 2008
Unix Commands: grep/sed/awk
Index
1. grep
The grep command globally searches for regular expressions in files and prints all lines that contain the expression.
The name of grep comes from the "globally search and print" command in the ex editor:
:g/pattern/p
Since patterns are also called "regular expressions", the above command becomes:
:g/re/p
Thus grep.
Exit Status
The exit status of grep is 0 if the matching succeeded, 1 if the pattern was not found, and 2 if the file was not found.
Regular Expression Metacharacters for grep (also for vi, ex, sed, and awk)
^ | beginning of line |
$ | end of line |
. | one character |
* | zero or more characters |
[] | any one in the set |
[^] | not in the set |
\< | beginning of word, e.g. \ |
\> | end of word, e.g. love\> matches words that ends with love |
\(pattern\) | remember the matched pattern; up to 9 can be remembered; recall with \1, \2, etc. |
x\{m\} | repetition of character x for exactly m times |
x\{m,\} | repetition of character x for at least m times |
x\{m,n\} | repetition of character x for at least m times and no more than n times |
Example:
% cat sample
northwest NW
northeast NE
north N
% grep '\# match all lines containing words
# starting with "north"
northwest NW
northeast NE
north N
% grep '\' sample
# match all lines containing the word "north"
# this is the same as grep -w north sample
north N
2. sed (Streamlined Editor)
The sed editor is not an interactive editor like vi. It executes editing commands on a file and prints the result to standard output. It is nondestructive, meaning the original file is not changed in any way. sed commands can also be put in a script file.
Template:
sed 'command' filename
sed processes one line at a time. The line being processed is stored in a temporary buffer called a pattern space.
Exit Status
The exit status of sed is zero for success, and nonzero otherwise.
Line Addresses
Line addressing determines which lines will be processed. An address consists of one number or two numbers separated by a comma (,), or a regular expression.
Example:
% sed '1,3d' file # delete line 1 to 3 in file
% sed '/good/d' file # delete all lines containing "good"
Regular Expression Metacharacters
sed use the same metacharacters as grep, plus "&" wich remembers the search pattern.
Example:
s/love/**&** # changes love to **love**
sed Commands
a\ | appends one or more lines of text to the current line |
c | changes text in the current line with new text |
d | deletes lines |
i\ | inserts text above the current line |
h | copies the contents of the pattern space to a holding buffer |
H | appends the contents of the pattern space to a holding buffer |
g | gets what is in the holding buffer and copies into the pattern buffer, overwriting what was there |
G | gets what is in the holding buffer and copies into the pattern buffer, appending to what was there |
l | lists nonprinting characters |
p | prints lines |
n | reads the next input line and starts processing the new line with the next command rather than the first command |
q | quits or exits |
r | reads line from a file |
! | applies the command to all lines except the selected ones |
Substitution Flags
s | substitutes one string for another |
g | global substitution on one line |
p | prints line |
w | writes lines to a file |
x | exchanges contents of the holding buffer with the pattern space |
y | translates one character to another |
sed Options
-e | allows multiple edits |
-n | suppresses default output |
-f | reads a sed script file |
Examples:
The p Command
% sed '/north/p' file # prints all lines matching north, then prints
# all the lines in the file since by default
# sed prints all lines.
% sed -n '/north/p' file # prints lines matching north
% sed -n '/west/, /east/p' file # all lines between the 2
# patterns are printed
The d Command
% sed '/north/d' file # deletes all lines matching north
The -e option allows multiple edits:
% sed -e '1,3d' -e 's/ABC/XYZ/'file # deletes lines 1 to 3, then
# replaces ABC with XYZ
The r Command and the w Command
% sed '/ABC/r file2' file # read the contents of file2 and insert
# it after each of the lines in file
# that matches ABC
% sed -n '/ABC/w file2' file # writes all lines in file that match
# ABC to file2
The a\ Command and the i\ Command
% sed '/ABC/a\This is a new line.' file # appends the new line into
# file after where ABC is matched
% sed '/ABC/i\This is a new line.' file # inserts the new line into
# file before where ABC is matched
The n Command
% sed '/ABC/{n; s/bad/good/;}' file # find the line that matches ABC,
# then replace bad with good on the next line
The q Command
% sed '5q' file # print the first 5 lines and then quit
Holding and Getting
% sed -e '/ABC/h' -e '$G' file
# copy the line matching ABC into the
# holding buffer, and then paste it to the end of
# the file. This is similar to the yank and paste
# commands of vi.
% sed -e '/ABC/{h;d;}' -e '/XYZ/{G;}' file
# copy the line matching
# ABC into the holding buffer, and then delete it;
# then paste it after the line matching XYZ.
# This is similar to the move command of vi.
3. awk
3.1 Basics
awk is a utility used for manipulating textes and generating reports. There are a number versions of awk: the old awk (awk), the new awk (nawk), and the GNU awk (gawk).
Syntax
awk 'command' file
The most general form of the command is as follows:
awk 'BEGIN {initializations}
search_pattern1 {actions}
search_pattern2 {actions}
...
END {final actions}' file
Of course it has to be on the same line, or put in a script.
- Commands in the initialization part are run once. Then for each line of the input, the search patterns are matched one by one, and corresponding actions taken if the pattern matches. Finally, the actions in the final actions part are taken.
- Search patterns are put between a pair of /
- If a search pattern is missing, then the corresponding actions will be taken for each line.
- If action is missing, the default is to print the entire line.
The following simple commands are used most often.
nawk 'pattern' file # print lines matching pattern
nawk '{action}' file # take action for every line
nawk 'pattern {action}' file
Records and fields
A line is considered to have a number of fields. By default field separators are spaces.
awk sees lines of input files as records. By default, the output and input records separators are a carriage return, stored in the built-in variables ORS and RS, respectively.$0 refers to the entire record (line). The line (record) number is stored in the built-in variable NR.
% nawk '{print NR, $0} file # precede each line with line number
Each record consists of fields which, by default, are separated by white space. The number of fields is kept in the built-in variable NF. The fields are represented by $1, $2, etc.
Field separators can be reset with the -F option.
% nawk -F: '/ABC/{print $1, $2}' file # set field separator to :
% nawk -F'[ :\t]' '{print $1, $2}' file
# set field separator to space :, and \t
- $1, $2, $3, ... represent the 1st, 2nd, 3rd field, and so on.
- $0 represents the entire line.
Examples
% nawk '/ABC/' file # prints lines matching ABC
% nawk '{print $1} file # prints the first field of file
% date
Sat Oct 24 12:34 EDT 2003
% date | nawk '{print "Month: " $2 "\nYear: ", $6}'
Month: Oct
Year: 2003
% date | nawk '{print "Month: " $2 "\nYear: " $6}'
Month: Oct
Year: 2003
3.2 Search patterns
A pattern consists of a regular expression, a boolean expression, or a combination of both. A pattern is put between a pair of "/".
Regular expressions
Regular expressions can be used as patterns.
Regular Expression Metacharacters
^ | beginning of line |
$ | end of line |
. | single character |
* | zero or more of preceding character |
+ | one or more of preceding character |
? | zero or one of preceding character |
[ABC] | any one in the set |
[^ABC] | any one not in the set |
[A-Z] | any one in the range |
[A|B] | A or B |
(AB)+ | one or more sets of AB |
\* | literal * |
& | remember the pattern |
The Match Operator ~ and the Not Operator !
Search can be constrained to a single field with the match operator. For example, $1 ~ /ABC/
matches 1st field against the pattern ABC
.
Examples
% nawk '$1 ~ /ABC/' file # print lines whose 1st field matches ABC
% nawk '$1 !~ /ABC/' file # print lines whose 1st field doesn't
# match ABC
It is possible to search for a block of consecutive lines using one search pattern to match the first line in the block and another pattern to match the last line in the block. For example:
/ABC/,/XYZ/
matches a block whose first line contain the pattern ABC
and last line contains XYZ
.
Relational and Logical Operations
awk fields can serve as operands of relational, arithmetic, and logical operators (same as C operators).Example
% nawk '$2 == 1234' file
# print lines whose 2nd field is equal to 1234
% nawk '{max=($1 > $2) ? $1 : $2; print max} file
# for each line print the max of the 1st and 2nd fields
% nawk '$2 > 5 && $2 < 15' file
# print lines whose 2nd field is betwee 5 and 15
% nawk 'NR==2, NR==4 {print $0}' file
# print lines 2 to 4
Here is a trick to test if field 1 is a numeric value.
(( $1 + 0 ) == $1 )If it is a string, the left-hand side would be 0, and the equality would be false.
3.3 Variables
Variables don't require declaration; actually there's no variable declaration, although variables should probably be initialized in the BEGIN
part.
Variables don't have data type, and can be used to store either string or numeric values. A text string that doesn't look like a number will be treated as 0 in a numeric operation. The following 2 lines are actually equivalent.
v = 100
v = "100"
An uninitialized variable has a value 0, but will print nothing.
Built-in variables
$0, $1, $2,... | Field variables |
FILENAME | Name of input file |
FS | Field separator, by default the space character. Can be modified. |
NF | Number of fields in the current record |
NR | Number of the current record (line number) |
OFMT | Output format (default: "%.6g") |
Field variables are not read-only.
% nawk 'BEGIN{OFMT="%.2f"; print 1.23456, 12E-2}' file
1.23 0.12
Arrays
Like variables, arrays require no declaration. Arrayes are one-dimensional and the first index is 1. Strings can be used as array index, essentially creating a dictionary.3.4 Operators
awk supports C arithmetic and assignment operators. For strings, it supports concatenation.% awk 'BEGIN { x=3; y=x+2; print y}'
% awk 'BEGIN { s="hello " "world"; print s} # prints "hello world"
3.5 Built-in functions
length()
The length() function returns the length of its parameter as a string. If no parameter is given (in this case, () are not necessary), it returns the length of the input line.
sqrt(), exp(), log(), int()(get the integer part of the parameter)
substr()
The substr(string, start, max size of substring) function returns a substring.
split()
The split(string, array, [field separator]) function splits a string into fields and store the fields in an array.
index()
The index(string, pattern) function returns the starting index of a pattern in a string; 0 if the pattern is not matched.
print()
- print() by itself prints the input line.
- print() with one argument prints the argument.
- print() with multiple arguments prints all of them, separated by whitespace (or as specified by OFS) when the arguments are separated by commas, or concatenated when the arguments are separated by spaces.
Commas between the arguments of the print command will be printed as OFS (ouput field separator, which by default is a white space).
% cat file
ABC:XYZ
% nawk -F: '{print $1, $2}' file
ABC XYZ
% nawk -F: '{print $1 $2}' file
ABCXYZ
printf()
The printf() function is similar to the function in C.
% echo "UNIX" | nawk '{printf "|%15s|\n", $1}'
| UNIX|
3.6 Control structures
awk supports C control structures: if/else, for-loop, while-loop.
3.7 Recipes
Precede each line with a line number
% nawk '{print NR, $0}' file
Add numbers in a column
% cat file
0.1
0.2
0.3
% nawk '{t += $1} END {print t}' file
0.6
Add numbers with same index
In the following file, the first column are index, and we are to add numbers with the same index.
% cat file
1 0.1
2 0.2
3 0.3
1 40
2 50
3 60
% nawk '{a[$1] += $2} END {for(i=1; i<=3; i++) print a[i]}' file
40.1
50.2
60.3
For each line matching a pattern prints its previous line
% cat file
some line
line before pattern
the pattern ABC here
some line
another line before pattern
pattern ABC again
more line
% nawk '/ABC/ {print pre} {pre=$0}' file
line before pattern
another line before pattern
Connecting to MySQL
August 15, 2008
Ubuntu LAMP Server - Server Updates
1. Create a backup of the source.list file
> cp /etc/apt/sources.list /etc/apt/sources.backup
2. Edit the source.list file
> nano /etc/apt/sources.list
-> Uncomment the universe and multiverse repositories
-> Save the file
3. Update the package list
> sudo apt-get update
August 14, 2008
MySQL - SQL Statements
Selecting a database:
mysql> USE database;
Listing databases:
mysql> SHOW DATABASES;
Listing tables in a db:
mysql> SHOW TABLES;
Describing the format of a table:
mysql> DESCRIBE table;
Creating a database:
mysql> CREATE DATABASE db_name;
Creating a table:
mysql> CREATE TABLE table_name (field1_name TYPE(SIZE), field2_name TYPE(SIZE));
Ex: mysql> CREATE TABLE pet (name VARCHAR(20), sex CHAR(1), birth DATE);
Load tab-delimited data into a table:
mysql> LOAD DATA LOCAL INFILE "infile.txt" INTO TABLE table_name;
(Use \n for NULL)
Inserting one row at a time:
mysql> INSERT INTO table_name VALUES ('MyName', 'MyOwner', '2002-08-31');
(Use NULL for NULL)
Retrieving information (general):
mysql> SELECT from_columns FROM table WHERE conditions;
All values: SELECT * FROM table;
Some values: SELECT * FROM table WHERE rec_name = "value";
Multiple critera: SELECT * FROM TABLE WHERE rec1 = "value1" AND rec2 = "value2";
Reloading a new data set into existing table:
mysql> SET AUTOCOMMIT=1; # used for quick recreation of table
mysql> DELETE FROM pet;
mysql> LOAD DATA LOCAL INFILE "infile.txt" INTO TABLE table;
Fixing all records with a certain value:
mysql> UPDATE table SET column_name = "new_value" WHERE record_name = "value";
Selecting specific columns:
mysql> SELECT column_name FROM table;
Retrieving unique output records:
mysql> SELECT DISTINCT column_name FROM table;
Sorting:
mysql> SELECT col1, col2 FROM table ORDER BY col2;
Backwards: SELECT col1, col2 FROM table ORDER BY col2 DESC;
Date calculations:
mysql> SELECT CURRENT_DATE, (YEAR(CURRENT_DATE)-YEAR(date_col)) AS time_diff [FROM table];
MONTH(some_date) extracts the month value and DAYOFMONTH() extracts day.
Pattern Matching:
mysql> SELECT * FROM table WHERE rec LIKE "blah%";
(% is wildcard - arbitrary # of chars)
Find 5-char values: SELECT * FROM table WHERE rec like "_____";
(_ is any single character)
Extended Regular Expression Matching:
mysql> SELECT * FROM table WHERE rec RLIKE "^b$";
(. for char, [...] for char class, * for 0 or more instances
^ for beginning, {n} for repeat n times, and $ for end)
(RLIKE or REGEXP)
To force case-sensitivity, use "REGEXP BINARY"
Counting Rows:
mysql> SELECT COUNT(*) FROM table;
Grouping with Counting:
mysql> SELECT owner, COUNT(*) FROM table GROUP BY owner;
(GROUP BY groups together all records for each 'owner')
Selecting from multiple tables:
(Example)
mysql> SELECT pet.name, comment FROM pet, event WHERE pet.name = event.name;
(You can join a table to itself to compare by using 'AS')
Currently selected database:
mysql> SELECT DATABASE();
Maximum value:
mysql> SELECT MAX(col_name) AS label FROM table;
Auto-incrementing rows:
mysql> CREATE TABLE table (number INT NOT NULL AUTO_INCREMENT, name CHAR(10) NOT NULL);
mysql> INSERT INTO table (name) VALUES ("tom"),("dick"),("harry");
Removing a column:
mysql> ALTER TABLE tbl DROP COLUMN col;
(Full ALTER TABLE syntax available at mysql.com.)
Batch mode (feeding in a script):
# mysql -u user -p <> source batch_file;
Backing up a database with mysqldump:
# mysqldump --opt -u username -p database > database_backup.sql
(More info at MySQL's docs.)
Adding a new column to the existing table
MySQL> Alter Table [table name] Add [New_Column] VARCHAR(Size) AFTER [Column_Name];
Adding a new column and put the column first
MySQL> Alter Table [table name] Add [New_Column] VARCHAR(Size) FIRST;
Ubuntu LAMP Server - MySQL Configuration
1. MySQL Bind Address
The purpose of this configuration is to allow remote clients to access the database. If this is not set properly, an error message "Can not connect to mysql error 10061", will be retrieved when trying to access.
> nano /etc/mysql/my.conf
-> Change the line: Bind-address = localhost to your IP address or comment out to automatically retain the host address
2. Set MySQL root password
For security purposes and best practices is always best to change the root password.
> mysql -u root -p (log into the db)
> SET PASSWORD FOR 'root' @ 'localhost' = PASSWORD ('yourpassword');
3. Create your MySQL db
> CREATE DATABASE mydb; (mydb is your db name)
4. Create a user with all privileges
> GRANT ALL PRIVILEGES ON *.* TO 'root' @ 'localhost' IDENTIFIED BY 'yourpwd' WITH GRANT OPTION;
-> The above statement gives the root account all privileges to the all tables in the databases
5. Create a MySQL user with fewer priviliges
> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORY TABLES, LOCK TABLES ON yourdb.* TO 'root' @ 'ip address" IDENTIFIED BY 'yourpwd';
-> This command gives the the user root fewer privileges to all tables in db 'your db'
There are MySQL admin tools that will allow you to administer the database from another node.
August 13, 2008
Ubuntu LAMP Server Installation
> sudo nano /etc/network/interfaces
-> modified the interfaces with the following setting
-> network address
-> net mask
-> ip address
-> gateway
I ran into a couple of issues. The main issue had to do with the server communicating with other host on the LAN. The setup include the Ubuntu LAMP Server, one laptop, and a hub. Initially, I couldn't ping the local host address nor the laptop. The local host address could not be scanned because the configuration file (interfaces) was misconfigured.
Issue: Unable to ping local host
Solution: Configure and ensure the /etc/network/interfaces file contains the following comments:
# The loopback interface
auto lo
iface lo inet loopback
Issue: Unable to ping remote host on LAN
Solution: