Linux command line reference

This is a Linux command line reference for common operations. Examples marked with • are valid/safe to paste without modification into a terminal, so you may want to keep a terminal window open while reading this so you can cut & paste.

Commands
•apropos whatis
Show commands pertinent to string. See also threadsafe

•man -t man | ps2pdf - > man.pdf
make a pdf of a manual page

which command
Show full path name of command

time command
See how long a command takes

•time cat
Start stopwatch. Ctrl-d to stop. See also sw

•nice info
Run a low priority command (The "info" reader in this case)

•renice 19 -p $$
Make shell (script) low priority. Use for non interactive tasks

dir navigation

•cd -
Go to previous directory

•cd
Go to $HOME directory

(cd dir && command)
Go to dir, execute command and return to current dir

•pushd .
Put current dir on stack so you can popd back to it

file searching

•alias l='ls -l --color=auto'
quick dir listing

•ls -lrt
List files by date. See also newest and find_mm_yyyy

•ls /usr/bin | pr -T9 -W$COLUMNS
Print in 9 columns to width of terminal

find -name '*.[ch]' | xargs grep -E 'expr'
Search 'expr' in this dir and below. See also findrepo

find -type f -print0 | xargs -r0 grep -F 'example'
Search all regular files for 'example' in this dir and below

find -maxdepth 1 -type f | xargs grep -F 'example'
Search all regular files for 'example' in this dir

find -maxdepth 1 -type d | while read dir; do echo $dir; echo cmd2; done
Process each item with multiple commands (in while loop)

•find -type f ! -perm -444
Find files not readable by all (useful for web site)

•find -type d ! -perm -111
Find dirs not accessible by all (useful for web site)

•locate -r 'file[^/]*\.txt'
Search cached index for names. This re is like glob *file*.txt

•look reference
Quickly search (sorted) dictionary for prefix

•grep --color reference /usr/share/dict/words
Highlight occurances of regular expression in dictionary

archives and compression

gpg -c file
Encrypt file

gpg file.gpg
Decrypt file

tar -c dir/ | bzip2 > dir.tar.bz2
Make compressed archive of dir/

bzip2 -dc dir.tar.bz2 | tar -x
Extract archive (use gzip instead of bzip2 for tar.gz files)

tar -c dir/ | gzip | gpg -c | ssh user@remote 'dd of=dir.tar.gz.gpg'
Make encrypted archive of dir/ on remote machine

find dir/ -name '*.txt' | tar -c --files-from=- | bzip2 > dir_txt.tar.bz2
Make archive of subset of dir/ and below

find dir/ -name '*.txt' | xargs cp -a --target-directory=dir_txt/ --parents
Make copy of subset of dir/ and below

( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p )
Copy (with permissions) copy/ dir to /where/to/ dir

( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p )
Copy (with permissions) contents of copy/ dir to /where/to/

( tar -c /dir/to/copy ) | ssh -C user@remote 'cd /where/to/ && tar -x -p'
Copy (with permissions) copy/ dir to remote:/where/to/ dir

dd bs=1M if=/dev/sda | gzip | ssh user@remote 'dd of=sda.gz'
Backup harddisk to remote machine

rsync (Network efficient file copier: Use the --dry-run option for testing)

rsync -P rsync://rsync.server.com/path/to/file file
Only get diffs. Do multiple times for troublesome downloads

rsync --bwlimit=1000 fromfile tofile
Locally copy with rate limit. It's like nice for I/O

rsync -az -e ssh --delete ~/public_html/ remote.com:'~/public_html'
Mirror web site (using compression and encryption)

rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/
Synchronize current directory with remote one

ssh (Secure SHell)

ssh $USER@$HOST command
Run command on $HOST as $USER (default command=shell)

•ssh -f -Y $USER@$HOSTNAME xeyes
Run GUI command on $HOSTNAME as $USER

scp -p -r $USER@$HOST: file dir/
Copy with permissions to $USER's home directory on $HOST

ssh -g -L 8080:localhost:80 root@$HOST
Forward connections to $HOSTNAME:8080 out to $HOST:80

ssh -R 1434:imap:143 root@$HOST
Forward connections from $HOST:1434 in to imap:143

wget (multi purpose download tool)

•(cd cli && wget -nd -pHEKk http://www.pixelbeat.org/cmdline.html)
Store local browsable version of a page to the current dir

wget -c http://www.example.com/large.file
Continue downloading a partially downloaded file

wget -r -nd -np -l1 -A '*.jpg' http://www.example.com/dir/
Download a set of files to the current directory

wget ftp://remote/file[1-9].iso/
FTP supports globbing directly

•wget -q -O- http://www.pixelbeat.org/timeline.html | grep 'a href' | head
Process output directly

echo 'wget url' | at 01:00
Download url at 1AM to current dir

wget --limit-rate=20k url
Do a low priority download (limit to 20KB/s in this case)

wget -nv --spider --force-html -i bookmarks.html
Check links in a file

wget --mirror http://www.example.com/
Efficiently update a local copy of a site (handy from cron)

networking (Note ifconfig, route, mii-tool, nslookup commands are obsolete)

ethtool eth0
Show status of ethernet interface eth0

ethtool --change eth0 autoneg off speed 100 duplex full
Manually set ethernet interface speed

iwconfig eth1
Show status of wireless interface eth1

iwconfig eth1 rate 1Mb/s fixed
Manually set wireless interface speed

•iwlist scan
List wireless networks in range

•ip link show
List network interfaces

ip link set dev eth0 name wan
Rename interface eth0 to wan

ip link set dev eth0 up
Bring interface eth0 up (or down)

•ip addr show
List addresses for interfaces

ip addr add 1.2.3.4/24 brd + dev eth0
Add (or del) ip and mask (255.255.255.0)

•ip route show
List routing table

ip route add default via 1.2.3.254
Set default gateway to 1.2.3.254

•tc qdisc add dev lo root handle 1:0 netem delay 20msec
Add 20ms latency to loopback device (for testing)

•tc qdisc del dev lo root
Remove latency added above

•host pixelbeat.org
Lookup DNS ip address for name or vice versa

•hostname -i
Lookup local ip address (equivalent to host `hostname`)

•whois pixelbeat.org
Lookup whois info for hostname or ip address

•netstat -tupl
List internet services on a system

•netstat -tup
List active connections to/from system

windows networking (Note samba is the package that provides all this windows specific networking support)

•smbtree
Find windows machines. See also findsmb

nmblookup -A 1.2.3.4
Find the windows (netbios) name associated with ip address

smbclient -L windows_box
List shares on windows machine or samba server

mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share
Mount a windows share

echo 'message' | smbclient -M windows_box
Send popup to windows machine (off by default in XP sp2)

text manipulation (Note sed uses stdin and stdout. Newer versions support inplace editing with the -i option)

sed 's/string1/string2/g'
Replace string1 with string2

sed 's/\(.*\)1/\12/g'
Modify anystring1 to anystring2

sed '/ *#/d; /^ *$/d'
Remove comments and blank lines

sed ':a; /\\$/N; s/\\\n//; ta'
Concatenate lines with trailing \

sed 's/[ \t]*$//'
Remove trailing spaces from lines

sed 's/\([\\`\\"$\\\\]\)/\\\1/g'
Escape shell metacharacters active within double quotes

•seq 10 | sed "s/^/      /; s/ *\(.\{7,\}\)/\1/"
Right align numbers

sed -n '1000p;1000q'
Print 1000th line

sed -n '10,20p;20q'
Print lines 10 to 20

sed -n 's/.*<title>\(.*\)<\/title>.*/\1/ip;T;q'
Extract title from HTML web page

sed -i 42d ~/.ssh/known_hosts
Delete a particular line

sort -t. -k1,1n -k2,2n -k3,3n -k4,4n
Sort IPV4 ip addresses

•echo 'Test' | tr '[:lower:]' '[:upper:]'
Case conversion

•tr -dc '[:print:]' < /dev/urandom
Filter non printable characters

•history | wc -l
Count lines

set operations (Note you can export LANG=C for speed. Also these assume no duplicate lines within a file)

sort file1 file2 | uniq
Union of unsorted files

sort file1 file2 | uniq -d
Intersection of unsorted files

sort file1 file1 file2 | uniq -u
Difference of unsorted files

sort file1 file2 | uniq -u
Symmetric Difference of unsorted files

join -a1 -a2 file1 file2
Union of sorted files

join file1 file2
Intersection of sorted files

join -v2 file1 file2
Difference of sorted files

join -v1 -v2 file1 file2
Symmetric Difference of sorted files

math

•echo '(1 + sqrt(5))/2' | bc -l
Quick math (Calculate φ). See also bc

•echo 'pad=20; min=64; (100*10^6)/((pad+min)*8)' | bc
More complex (int) e.g. This shows max FastE packet rate

•echo 'pad=20; min=64; print (100E6)/((pad+min)*8)' | python
Python handles scientific notation

•echo 'pad=20; plot [64:1518] (100*10**6)/((pad+x)*8)' | gnuplot -persist
Plot FastE packet rate vs packet size

•echo 'obase=16; ibase=10; 64206' | bc
Base conversion (decimal to hexadecimal)

•echo $((0x2dec))
Base conversion (hex to dec) ((shell arithmetic expansion))

•units -t '100m/9.69s' 'miles/hour'
Unit conversion (metric to imperial)

•units -t '500GB' 'GiB'
Unit conversion (SI to IEC prefixes)

•units -t '1 googol'
Definition lookup

•seq 100 | (tr '\n' +; echo 0) | bc
Add a column of numbers. See also add and funcpy

calendar

•cal -3
Display a calendar

•cal 9 1752
Display a calendar for a particular month year

•date -d fri
What date is it this friday. See also day

•[ $(date -d "tomorrow" +%d) = "01" ] || exit
exit a script unless it's the last day of the month

•date --date='25 Dec' +%A
What day does xmas fall on, this year

•date --date='@2147483647'
Convert seconds since the epoch (1970-01-01 UTC) to date

•TZ=':America/Los_Angeles' date
What time is it on West coast of US (use tzselect to find TZ)

echo "mail -s 'get the train' P@draigBrady.com < /dev/null" | at 17:45
Email reminder

•echo "DISPLAY=$DISPLAY xmessage cooker" | at "NOW + 30 minutes"
Popup reminder

locales

•printf "%'d\n" 1234
Print number with thousands grouping appropriate to locale

•BLOCK_SIZE=\'1 ls -l
get ls to do thousands grouping appropriate to locale

•echo "I live in `locale territory`"
Extract info from locale database

•LANG=en_IE.utf8 locale int_prefix
Lookup locale info for specific country. See also ccodes

•locale | cut -d= -f1 | xargs locale -kc | less
List fields available in locale database

recode (Obsoletes iconv, dos2unix, unix2dos)

•recode -l | less
Show available conversions (aliases on each line)

recode windows-1252.. file_to_change.txt
Windows "ansi" to local charset (auto does CRLF conversion)

recode utf-8/CRLF.. file_to_change.txt
Windows utf8 to local charset

recode iso-8859-15..utf8 file_to_change.txt
Latin9 (western europe) to utf8

recode ../b64 < file.txt > file.b64
Base64 encode

recode /qp.. < file.txt > file.qp
Quoted printable decode

recode ..HTML < file.txt > file.html
Text to HTML

•recode -lf windows-1252 | grep euro
Lookup table of characters

•echo -n 0x80 | recode latin-9/x1..dump
Show what a code represents in latin-9 charmap

•echo -n 0x20AC | recode ucs-2/x2..latin-9/x
Show latin-9 encoding

•echo -n 0x20AC | recode ucs-2/x2..utf-8/x
Show utf-8 encoding

CDs

gzip < /dev/cdrom > cdrom.iso.gz
Save copy of data cdrom

mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz
Create cdrom image from contents of dir

mount -o loop cdrom.iso /mnt/dir
Mount the cdrom image at /mnt/dir (read only)

cdrecord -v dev=/dev/cdrom blank=fast
Clear a CDRW

gzip -dc cdrom.iso.gz | cdrecord -v dev=/dev/cdrom -
Burn cdrom image (use dev=ATAPI -scanbus to confirm dev)

cdparanoia -B
Rip audio tracks from CD to wav files in current dir

cdrecord -v dev=/dev/cdrom -audio *.wav
Make audio CD from all wavs in current dir (see also cdrdao)

oggenc --tracknum='track' track.cdda.wav -o 'track.ogg'
Make ogg file from wav file

disk space (See also FSlint)

•ls -lSr
Show files by size, biggest last

•du -s * | sort -k1,1rn | head
Show top disk users in current dir. See also dutop

•df -h
Show free space on mounted filesystems

•df -i
Show free inodes on mounted filesystems

•fdisk -l
Show disks partitions sizes and types (run as root)

•rpm -q -a --qf '%10{SIZE}\t%{NAME}\n' | sort -k1,1n
List all packages by installed size (Bytes) on rpm distros

•dpkg-query -W -f='${Installed-Size;10}\t${Package}\n' | sort -k1,1n
List all packages by installed size (KBytes) on deb distros

•dd bs=1 seek=2TB if=/dev/null of=ext3.test
Create a large test file (taking no space). See also truncate

monitoring/debugging

•tail -f /var/log/messages
Monitor messages in a log file

•strace -c ls >/dev/null
Summarise/profile system calls made by command

•strace -f -e open ls >/dev/null
List system calls made by command

•ltrace -f -e getenv ls >/dev/null
List library calls made by command

•lsof -p $$
List paths that process id has open

•lsof ~
List processes that have specified path open

•tcpdump not port 22
Show network traffic except ssh. See also tcpdump_not_me

•ps -e -o pid,args --forest
List processes in a hierarchy

•ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed '/^ 0.0 /d'
List processes by % cpu usage

•ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS
List processes by mem usage. See also ps_mem.py

•ps -C firefox-bin -L -o pid,tid,pcpu,state
List all threads for a particular process

•ps -p 1,2
List info for particular process IDs

•last reboot
Show system reboot history

•free -m
Show amount of (remaining) RAM (-m displays in MB)

•watch -n.1 'cat /proc/interrupts'
Watch changeable data continuously

system information (see also sysinfo) ('#' means root access is required)

•uname -a
Show kernel version and system architecture

•head -n1 /etc/issue
Show name and version of distribution

•cat /proc/partitions
Show all partitions registered on the system

•grep MemTotal /proc/meminfo
Show RAM total seen by the system

•grep "model name" /proc/cpuinfo
Show CPU(s) info

•lspci -tv
Show PCI info

•lsusb -tv
Show USB info

•mount | column -t
List mounted filesystems on the system (and align output)

•grep -F capacity: /proc/acpi/battery/BAT0/info
Show state of cells in laptop battery

#dmidecode -q | less
Display SMBIOS/DMI information

#smartctl -A /dev/sda | grep Power_On_Hours
How long has this disk (system) been powered on in total

#hdparm -i /dev/sda
Show info about disk sda

#hdparm -tT /dev/sda
Do a read speed test on disk sda

#badblocks -s /dev/sda
Test for unreadable blocks on disk sda

interactive (see also linux keyboard shortcuts)

•readline
Line editor used by bash, python, bc, gnuplot, ...

•screen
Virtual terminals with detach capability, ...

•mc
Powerful file manager that can browse rpm, tar, ftp, ssh, ...

•gnuplot
Interactive/scriptable graphing

•links
Web browser

•xdg-open http://www.pixelbeat.org/
open a file or url with the registered desktop application

miscellaneous

•alias hd='od -Ax -tx1z -v'
Handy hexdump. (usage e.g.: • hd /proc/self/cmdline | less)

•alias realpath='readlink -f'
Canonicalize path. (usage e.g.: • realpath ~/../$USER)

•set | grep $USER
Search current environment

touch -c -t 0304050607 file
Set file timestamp (YYMMDDhhmm)

•python -m SimpleHTTPServer
Serve current directory tree at http://$HOSTNAME:8000/

Super fast Database Copying/Cloning

A database cloning procedure is especially useful for the DBA who wants to give his developers a full-sized TEST and DEV instance by cloning the PROD instance into the development server areas.

This Oracle clone procedure can be use to quickly migrate a system from one UNIX server to another.  It clones the Oracle database and this Oracle cloning procedures is often the fastest way to copy a Oracle database.

STEP 1: On the old system, go into SQL*Plus, sign on as SYSDBA and issue: “alter database backup controlfile to trace”. This will put the create database syntax in the trace file directory. The trace keyword tells oracle to generate a script containing a create controlfile command and store it in the trace directory identified in the user_dump_dest parameter of the init.ora file. It will look something like this:

STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "OLDLSQ" NORESETLOGS
NOARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 2
MAXDATAFILES 240
MAXINSTANCES 1
MAXLOGHISTORY 113
LOGFILE
GROUP 1 ('/u03/oradata/oldlsq/log1a.dbf',
'/u03/oradata/olslsq/log1b.dbf') SIZE 30M,
GROUP 2 ('/u04/oradata/oldlsq/log2a.dbf',
'/u04/oradata/oldlsq/log2b.dbf') SIZE 30M
DATAFILE
'/u01/oradata/oldlsq/system01.dbf',
'/u01/oradata/oldlsq/mydatabase.dbf'
;
# Recovery is required if any of the datafiles are restored
# backups, or if the last shutdown was not normal or immediate.
RECOVER DATABASE
# Database can now be opened normally.
ALTER DATABASE OPEN;

STEP 2: Shutdown the old database
STEP 3: Copy all data files into the new directories on the new server. You may change the file names if you want, but you must edit the controlfile to reflect the new data files names on the new server.

rcp /u01/oradata/oldlsq/* newhost:/u01/oradata/newlsq
rcp /u01/oradata/oldlsq/* newhost:/u01/oradata/newlsq
rcp /u03/oradata/oldlsq/* newhost:/u03/oradata/newlsq
rcp /u04/oradata/oldlsq/* newhost:/u04/oradata/newlsq

STEP 4: Copy and Edit the Control file – Using the output syntax from STEP 1, modify the controlfile creation script by changing the following:
Old:

CREATE CONTROLFILE REUSE DATABASE "OLDLSQ" NORESETLOGS

New:

CREATE CONTROLFILE SET DATABASE "NEWLSQ" RESETLOGS

STEP 5: Remove the “recover database” and “alter database open” syntax

# Recovery is required if any of the datafiles are restored
# backups, or if the last shutdown was not normal or immediate.
RECOVER DATABASE
# Database can now be opened normally.
ALTER DATABASE OPEN;

STEP 6: Re-names of the data files names that have changed.

Save as db_create_controlfile.sql.
Old:

DATAFILE
'/u01/oradata/oldlsq/system01.dbf',
'/u01/oradata/oldlsq/mydatabase.dbf'

New:

DATAFILE
'/u01/oradata/newlsq/system01.dbf',
'/u01/oradata/newlsq/mydatabase.dbf'

STEP 7: Create the bdump, udump and cdump directories

cd $DBA/admin
mkdir newlsq
cd newlsq
mkdir bdump
mkdir udump
mkdir cdump
mkdir pfile

STEP 8: Copy-over the old init.ora file

rcp $DBA/admin/olslsq/pfile/*.ora newhost:/u01/oracle/admin/newlsq/pfile

STEP 9: Start the new database

@db_create_controlfile.sql

STEP 10: Place the new database in archivelog mode

SDLC Model – Overview of Phases:

1. Phase I - Project Initiation:
    a. Request for service – project justification
        i. Maintenance/enhancements/upgrade projects (minor in nature): Departments/agencies request directly to DoT representative using e-mail or memorandum.
        ii. New projects or larger system enhancements: Departments/agencies request to the Information Technology Committee (ITC) for IT staff resources and/or funding.
        iii. Feasibility Study: request sent to ITC for approval.
    b. Feasibility study and risk analysis
    c. Define Project Charter and identify project team and stakeholders
    d. Define scope and business objectives to achieve
    e. Assign IT personnel
    f. Develop project plan
    g. Assign project tracking number through DoT Request Tracking system

2. Phase II – Analysis and Requirements Definition
    a. Identify existing business processes and system flow
    b. Identify desired business processes and functional specifications
    c. Interview users (dept/agencies involved in project)
    d. Document requirements
    e. Identify interfaces with other systems (existing and new), internal and external to the organization
    f. Identify data conversion requirements
    g. Develop business rules and process flows
    h. Identify technical architecture
    i. Finalize project management plan
    j. Assign resources and schedule high-level tasks
    k. Obtain project sponsor(s) sign-off

3. Phase III – Request for Proposal (RFP) optional
a. Develop RFP document
    b. Identify potential vendors & deliver to Purchasing for processing
    c. Develop evaluation checklist
    d. Create vendor list and distribute responses to selection team for review
    e. Rank vendor responses; select top responders
    f. Schedule vendor demo’s
    g. Schedule site visits
    h. Contract negotiations
    i. Issue notice of intent to award
    j. Review and finalize vendor’s project plan and statement of work
    k. Develop final system cost including future maintenance agreement cost
    l. Final vendor selection with recommendation to ITC
    m. Contract signed by vendor and authorized city personnel
    n. Prepare purchase order (Service Contract)

4. Phase IV – Design
a. Design logical business model
    b. Design application components
    c. Design data dictionary
    d. Design security
    e. Design reports
    f. Design interfaces with external systems
    g. Design physical database/table structure
    h. Design technical architecture
    i. Obtain quotes and order hardware and software
    j. Develop user prototype and business case scenarios
    k. Design review/walk-through
    l. Obtain user sign-off

5. Phase V – Development
a. Create development and test environments
    b. Create development databases
    c. Develop graphical design and report layouts
    d. Define modules for coding and assign resources
    e. Create database roles and user security levels
    f. Create stored procedures and define database triggers
    g. Developer coding & documentation
    h. Quality Assurance of code
        i. Unit testing

6. Phase VI – Testing
a. Develop system test plans
    b. Train users involved in system testing
    c. System and user testing
    d. Obtain user sign-off
    e. Purchase and install production technical hardware/software
    f. Set up production security access
    g. Obtain user sign-off

7. Phase VII – Deployment
a. Create implementation plan
    b. Schedule IT Implementation Planning Session (all divisions of DoT)
    c. Complete documentation (system/technical) and training manuals
    d. Schedule and conduct end user training
    e. Move application components to production environment
    f. Deploy/move data to production environment (data conversion, import, or load data)
    g. User verification of data
    h. Review security
    i. Turnover operational processing to DoT Operations Center/user
    j. Final testing and verification of production environment
    k. Obtain user sign-off
    l. Go live with system

8. Phase VIII – Post Implementation Evaluation and Support
a. Conduct post implementation review
    b. Evaluate overall project delivery with DoT management
    c. Utilize DoT change management procedures for enhancements and incidents
    d. Monitor, support, and enhance system

Documents created in SDLC:
The following are documents which are created in entire SDLC…

1. User Requirements Specification
2. System Requirement Specification
3. System Design Document
4. Test Plans
5. Test Cases
6. Unified Implementation Plan

Using Fast Incremental Backups with Block Change Tracking in Oracle 10g

In Oracle 9i we could create incremental backups with level 0 to 4. A level 0 backup is a full backup.
In Oracle 10g there are still these levels but we only use incremental level 0 and 1 backups with Oracle’s suggested backup strategy.

Starting with Oracle 10g RMAN can take incremental backups without having to read the entire datafiles in order to find out which blocks have changed since the last backup.
This new feature is called FAST INCREMENTAL BACKUP.
The new technology used for this feature is called BLOCK CHANGE TRACKING.

You enbale block change tracking by making an entry in the controlfile:
SQL> alter database enable block change tracking;
alter database enable block change tracking
*
ERROR at line 1:
ORA-19773: must specify change tracking file name
If you do not have DB_CREATE_FILE_DEST specified (used for OMF => oracle managed files)
you will encounter an error if you do not specify a name fot the change tracking file
SQL> show parameter create
NAME TYPE VALUE
———————————— ———– ——————————
create_bitmap_area_size integer 8388608
create_stored_outlines string
db_create_file_dest string
db_create_online_log_dest_1 string
db_create_online_log_dest_2 string
db_create_online_log_dest_3 string
db_create_online_log_dest_4 string
db_create_online_log_dest_5 string
So I try to alter this parameter:
SQL> alter system set db_create_file_dest=’/home/oracle/file_create’;
alter system set db_create_file_dest=’/home/oracle/file_create’
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-01261: Parameter db_create_file_dest destination string cannot be
translated
ORA-01262: Stat failed on a file destination directory
Linux Error: 2: No such file or directory
Oracle checks if the location existes before it allows us to change this parameter!
SQL> ! mkdir -p /home/oracle/file_create
SQL> alter system set db_create_file_dest=’/home/oracle/file_create’;
System altered.

Now we can enable block change tracking
SQL> alter database enable block change tracking;
Database altered.
As an alternative you can specify a user managed filename if you do not want to use the parameter:
alter database enable block change tracking using file ‘/home/oracle/change.trc’;

Let’s take a look at the alert_log now:
alter database enable block change tracking
Thu May 18 06:39:14 2006
ORA-19773 signalled during: alter database enable block change tracking…
Thu May 18 06:39:46 2006
alter database enable block change tracking using file
‘/home/oracle/change.trc’
Thu May 18 06:39:46 2006
Block change tracking file is current.
Starting background process CTWR
CTWR started with pid=21, OS id=26871
Block change tracking service is active.
Thu May 18 06:39:47 2006
Completed: alter database enable block change tracking using file
‘/home/oracle/change.trc’

This is what oerr returns for this error:
[oracle@kr12-KL2 bdump]$ oerr ora 19773
19773, 00000, “must specify change tracking file name”
// *Cause: No file name was specified with the ALTER DATABASE ENABLE
// CHANGE TRACKING command, and the DB_CREATE_FILE_DEST parameter
// was not set.
// *Action: Either specify a file name, or set the DB_CREATE_FILE_DEST
// parameter.

Let’s take a look at the change tracking file:
[oracle@edchr3p8 oracle]$ cd file_create/
[oracle@edchr3p8 file_create]$ ll
total 4
drwxr-x— 3 oracle oinstall 4096 May 3 15:55 ORCL
[oracle@edchr3p8 file_create]$ cd ORCL/
[oracle@edchr3p8 ORCL]$ ll
total 4
drwxr-x— 2 oracle oinstall 4096 May 3 15:55 changetracking
[oracle@edchr3p8 ORCL]$ cd changetracking/
[oracle@edchr3p8 changetracking]$ ll
total 11348
-rw-r—– 1 oracle oinstall 11600384 May 3 15:55 o1_mf_25l66r5c_.chg
# this is an OMF change tracking file!

The change tracking file has an initial size of 10Mb.
How to monitor the change treacking file from inside Oracle:
SQL> desc v$block_change_tracking
Name Null? Type
—————————————– ——– —————————-
STATUS VARCHAR2(10)
FILENAME VARCHAR2(513)
BYTES NUMBER
SQL> select * from v$block_change_tracking;
STATUS FILENAME BYTES
———- ————————————————– ———-
ENABLED /home/oracle/change.trc 11599872
Now that we have enabled block change tracking a new background process called CTWR (change track writer) is started and it will automatically be started for every new instance.
[oracle@kr12-KL2 ~]$ ps -ef grep orcl
oracle 9531 1 0 May16 ? 00:00:00 ora_pmon_orcl
oracle 9533 1 0 May16 ? 00:00:02 ora_psp0_orcl
oracle 9535 1 0 May16 ? 00:00:00 ora_mman_orcl
oracle 9537 1 0 May16 ? 00:00:06 ora_dbw0_orcl
oracle 9539 1 0 May16 ? 00:00:16 ora_lgwr_orcl
oracle 9541 1 0 May16 ? 00:00:17 ora_ckpt_orcl
oracle 9543 1 0 May16 ? 00:00:25 ora_smon_orcl
oracle 9545 1 0 May16 ? 00:00:00 ora_reco_orcl
oracle 9547 1 0 May16 ? 00:00:03 ora_cjq0_orcl
oracle 9549 1 0 May16 ? 00:00:18 ora_mmon_orcl
oracle 9551 1 0 May16 ? 00:00:02 ora_mmnl_orcl
oracle 9554 1 0 May16 ? 00:00:00 ora_d000_orcl
oracle 9556 1 0 May16 ? 00:00:00 ora_s000_orcl
oracle 9560 1 0 May16 ? 00:00:00 ora_qmnc_orcl
oracle 9574 1 0 May16 ? 00:00:06 ora_q000_orcl
oracle 10746 1 0 May16 ? 00:00:00 ora_q001_orcl
oracle 26854 26850 0 06:38 ? 00:00:00 oracleorcl (DESCRIPTION=
(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 26871 1 0 06:39 ? 00:00:00 ora_ctwr_orcl
oracle 27778 1 0 06:50 ? 00:00:00 ora_j000_orcl
oracle 27780 27153 0 06:50 pts/0 00:00:00 grep orcl

CTWR will track addresses of blocks which have changed since the last backup in the change tracking file from now on.
RMAN can use this information for the next incremental backup. It will be able to find out which block must be written to the backupset by just reading the change tracking file.
RMAN will not have to read the entire datafiles into the SGA in order to find out which blocks must be backed up as it had to do before 10g.
This methode is much faster.

Now what happens if we lose the change tracking file or if it gets corrupted?
Let’s corrupt is:
[oracle@edchr3p8 changetracking]$ echo ‘hallo’ > o1_mf_25l66r5c_.chg
And now let us startup the database with a corrupt change tracking file!
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-19751: could not create the change tracking file
ORA-19750: change tracking file:
‘/home/oracle/file_create/ORCL/changetracking/o1_mf_25l66r5c_.chg’
ORA-27038: created file already exists
Additional information: 1
ORA-27047: unable to read the header block of file
Linux Error: 2: No such file or directory
Additional information: 1
Wed May 3 16:03:30 2006
CHANGE TRACKING is enabled for this database, but the
change tracking file can not be found. Recreating the file.
Oracle tries to automaqtically recreate the missing file but it cannot overwrite an existing on!

Let us create some changed blocks now so Oracle cannot track them in the change tracking file!

SQL> update hr.employees set salary=salary*1.1;
107 rows updated.
SQL> commit;
Commit complete.

Now let us take a look at the ALERT_LOG:

CHANGE TRACKING ERROR 19756, disabling change tracking
Wed May 3 16:07:24 2006
Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_ctwr_6628.trc:
ORA-19756: corrupt block number 18 found in change tracking file
ORA-19750: change tracking file: ‘/home/oracle/file_create/ORCL/changetracking/o1_mf_25l6qckh_.chg’
Block change tracking service stopping.
Deleted Oracle managed file /home/oracle/file_create/ORCL/
changetracking/o1_mf_25l6qckh_.chg
Oracle automatically deletes a currupted change tracking file and stops block change tracking.

Let us now take an incremental backup with RMAN:
RMAN> backup incremental level 1 for recover of copy database;
Starting backup at 03-MAY-06
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=159 devtype=DISK
channel ORA_DISK_1: starting incremental level 1 datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00001 name=/u01/app/oracle/oradata/orcl/system01.dbf
input datafile fno=00003 name=/u01/app/oracle/oradata/orcl/sysaux01.dbf
input datafile fno=00005 name=/u01/app/oracle/oradata/orcl/example01.dbf
input datafile fno=00002 name=/u01/app/oracle/oradata/orcl/undotbs01.dbf
input datafile fno=00004 name=/u01/app/oracle/oradata/orcl/users01.dbf
channel ORA_DISK_1: starting piece 1 at 03-MAY-06
channel ORA_DISK_1: finished piece 1 at 03-MAY-06
piece handle=/u01/app/oracle/flash_recovery_area/ORCL/backupset/
2006_05_03/ o1_mf_nnnd1_TAG20060503
T160745_25l6×2s1_.bkp tag= TAG20060503T160745
comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:05
channel ORA_DISK_1: starting incremental level 1 datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including current control file in backupset
including current SPFILE in backupset
channel ORA_DISK_1: starting piece 1 at 03-MAY-06
channel ORA_DISK_1: finished piece 1 at 03-MAY-06
piece handle=/u01/app/oracle/flash_recovery_area/ORCL/
backupset/2006_05_03/o1_mf_ncsn1_TAG20060503
T160745_25l6z4mv_.bkp tag= TAG20060503T160745
comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:02
Finished backup at 03-MAY-06

RMAN was able to take an incremental backup but now it had to use the old methode => read all the blocks in order to find out which ones have changed since the last backup;

Next I replace the change tracking file with an old version in order to see what happens when I try to take an incremental backup:

SYS @10gR2 SQL > ! mv /home/oracle/change.old /home/oracle/change.trc
RMAN> backup incremental level 1 for recover of copy database;
Starting backup at 19-MAY-06
using channel ORA_DISK_1
no parent backup or copy of datafile 3 found
no parent backup or copy of datafile 5 found
no parent backup or copy of datafile 6 found
no parent backup or copy of datafile 2 found
no parent backup or copy of datafile 4 found
channel ORA_DISK_1: starting incremental level 1 datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00001 name=/u01/app/oracle/oradata/orcl/system01.dbf
channel ORA_DISK_1: starting piece 1 at 19-MAY-06
channel ORA_DISK_1: finished piece 1 at 19-MAY-06
piece handle=/u01/app/oracle/flash_recovery_area/ORCL/backupset/2006_05_19/
o1_mf_nnnd1_TAG20060519T123227_26v7owgo_.bk p tag=
TAG20060519T123227 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile fno=00003 name=/u01/app/oracle/oradata/orcl/sysaux01.dbf
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/19/2006 12:32:30
ORA-19694: some changed blocks were not found in the change tracking file
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy
input datafile fno=00005 name=/u01/app/oracle/oradata/orcl/example01.dbf
output filename=/u01/app/oracle/flash_recovery_area/ORCL/datafile/
o1_mf_example_26v7oyjx_.dbf tag=TAG20060519T123227
recid=3 stamp=590848373
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile fno=00006 name=/u01/app/oracle/oradata/orcl/sales1.dbf
output filename=/u01/app/oracle/flash_recovery_area/ORCL/datafile/
o1_mf_sales_26v7pr8j_.dbf tag=TAG20060519T123227 rec id=4 stamp=590848386
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
channel ORA_DISK_1: starting datafile copy
input datafile fno=00002 name=/u01/app/oracle/oradata/orcl/undotbs01.dbf
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/19/2006 12:33:13
ORA-19694: some changed blocks were not found in the change tracking file
continuing other job steps, job failed will not be re-run
channel ORA_DISK_1: starting datafile copy
input datafile fno=00004 name=/u01/app/oracle/oradata/orcl/users01.dbf
output filename=/u01/app/oracle/flash_recovery_area/ORCL/datafile/
o1_mf_users_26v7qb8t_.dbf tag=TAG20060519T123227 rec id=5 stamp=590848394
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting incremental level 1 datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
including current control file in backupset
including current SPFILE in backupset
channel ORA_DISK_1: starting piece 1 at 19-MAY-06
channel ORA_DISK_1: finished piece 1 at 19-MAY-06
piece handle=/u01/app/oracle/flash_recovery_area/ORCL/backupset/2006_05_19/
o1_mf_ncsn1_TAG20060519T123227_26v7qfb5_.bk p tag=TAG20060519T123227 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================

RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/19/2006 12:33:13
ORA-19694: some changed blocks were not found in the change tracking file
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 05/19/2006 12:32:30
ORA-19694: some changed blocks were not found in the change tracking file
RMAN> list backup of tablespace sysaux;
There is no backup!!!
RMAN has found out that I tried to cheat and that it was not able to create an incremental backup for all files by using the change tracking file.
This is what oerr says about ORA-19694:
[oracle@lutzasm ~]$ oerr ora 19694
19694, 00000, “some changed blocks were not found in the change tracking file”
// *Cause: A backup or copy found that some changed blocks had not been
// recorded in the change tracking file. The details of which files
// and blocks are affected will be in an Oracle trace file.
// *Action: This indicates that there is a problem with the change tracking
// feature. Disable change tracking and re-start the backup.

connect lutz hartmann as sysdba