How to use VirtualBox with Vagrant

In the previous post, we saw how to use homebrew to install VirtualBox & Vagrant. In this post, we’ll try to use vagrant to spin up virtual development environment using VirtualBox.

  • Create a folder in your user home:

$ mkdir -p project/vagrant
$ cd ~/project/vagrant
$ pwd
/Users/xxxxx/project/vagrant

  • To add a vagrant box, type the following command:

$ vagrant box add ubuntu/trusty64
==> box: Loading metadata for box ‘ubuntu/trusty64’
box: URL: https://vagrantcloud.com/ubuntu/trusty64
==> box: Adding box ‘ubuntu/trusty64’ (v1.0.0) for provider: virtualbox
box: Downloading: https://vagrantcloud.com/hashicorp/boxes/trusty64/versions/1.0.0/providers/virtualbox.box
box: Progress: 5% (Rate: 100k/s, Estimated time remaining: 0:25:15)

Search here to find a box suitable for your need.

  • Once the box is successfully added, you can initialise the box:

$ vagrant init trusty64

It will generate a vagrantfile & .vagrant folder
$ ls -la
drwxr-xr-x 3 xxxxx staff 102 mmm 13 22:16 .vagrant
-rw-r–r–     1 xxxxx staff 3024 mmm 13 22:40 Vagrantfile

  • To fire up the box:

$ vagrant up
Bringing machine ‘default’ up with ‘virtualbox’ provider…
==> default: This machine used to live in /Users/juhfa/projects/chef but it’s now at /Users/juhfa/projects.
==> default: Depending on your current provider you may need to change the name of
==> default: the machine to run it as a different machine.
==> default: Clearing any previously set forwarded ports…
==> default: Clearing any previously set network interfaces…
==> default: Preparing network interfaces based on configuration…
default: Adapter 1: nat
==> default: Forwarding ports…
default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running ‘pre-boot’ VM customizations…
==> default: Booting VM…
==> default: Waiting for machine to boot. This may take a few minutes…

  • Once the machine is successfully started, you’ll be able to see it added in the VirtualBox GUI. By default, port 2222 of your local machine is used to connect to port 22 of the virtual machine
  • You can then ssh to the box using:

$vagrant ssh
Welcome to Ubuntu —–
———————–
vagrant@trusty64:~$

How to install Vagrant and VirtualBox on Mac

Vagrant

Vagrant is developed by Hashicorp, it is an useful tool for creating virtualised instances using Virtualbox, VMWare, AWS etc. With a simple vagrantfile, you can create uniform virtual development instances & you can also use vagrant instances to test Shell scripts, PowerShell scripts etc.  It is also very useful in testing your Configuration management cookbooks/modules.

For download & more details refer: https://www.vagrantup.com/

VirtualBox

Virtualbox is a free and awesome virtualisation solution by Oracle. You can spin up multiple instance on your Mac using it, it’s very useful when you want to spin up different Operating System(OS) instances on your Mac such as Ubuntu/Centos/Windows etc.

All you need is Operating System iso’s. Add the iso, set desired configuration and power up the instance. You will need to install the Operating System for first use, thereafter instance will be ready for use.

For download & more details refer: https://www.virtualbox.org

Installation of VirtualBox and Vagrant

Regular Method

  • Download the installer to your Mac using safari browser.
  • Double click on the installation package & follow the instructions to install VirtualBox and Vagrant

Homebrew

It’s said to be the missing package manager for MacOS, I can’t agree more. I found it really useful.

  • To install homebrew: Paste this command in your terminal/iterm
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • Add homebrew to your .bashrc/.bash_profile : export $PATH=$PATH:/usr/local/bin
  • To confirm installation is successful, Open a new terminal and type: brew doctor

Now we’ll use home-brew to install VirtualBox & Vagrant.

Go to terminal and type:

  • brew cask install virtualbox
  • brew cask install vagrant
  • brew cask install vagrant-manager

You should be able to see these applications in Launchpad.

Useful Linux Commands

Directories & Files

  1. To create multiple directories in one command & give it required permission

mkdir -pm 755 dir1/dir2/dir3

This command will create dir1, then create dir2 inside dir 1, then create dir 3 inside dir2 with 755 permissions on all dirs.

  1. To find the oldest file

ls -lt

ls -ltr → In reverse order

This command will sort the files by their modification time, or the last time it was changed.

  1. To find the newest file

ls -lu

  1. To find when was inode changed i.e. permission modification etc

ls -c

This command will list the files, as per the last changes in inode.

  1. To list all subdirectories

ls -R

  1. To list all directories

ls -l | grep ‘^d’

Alias ld=’ls -l | grep ‘^d’’

  1. To sort by Size

ls -lS

  1. To list the file with file types

ls -lAF

  1. To list the ending characters of the file

ls -l | cat -vte

cat -e option marks the end of lines with a $

  1. To list the file name in qoutes

ls -Q

  1. To list all hidden files

ls -la

  1. To list all hidden files except . & ..

ls -lA

  1. Counting files by type

find ${*-.} -type f -print | xargs file | awk ‘{ $1=NULL; t[$0]++; } END { for (i in t) printf (“%d\t%s\n”, t[i], i); }’ | sort -nr

Find command and it’s usage:

  1. To find all files in the present directory

find . -print

  1. why xargs option is used

When we give command ls -ld `find . -print` , it might give error, when command line is too long

hence we can give xargs

find . -print | xargs ls -ld

The generated command line length is the sum of the size, in bytes, of the Command and each Argument treated as strings, including a null byte terminator for each of these strings. The xargs command limits the command line length. When the constructed command line runs, the combinedArgument and environment lists can not exceed ARG_MAX bytes. Within this constraint, if you do not specify the -n or the -s flags, the default command line length is at least the value specified by LINE_MAX.

Examples

To use a command on files whose names are listed in a file, type:

xargs lint -a <cfiles

If the cfiles file contains the following text:

main.c readit.c
gettoken.c
putobj.c
the xargs command constructs and runs the following command:

lint -a main.c readit.c gettoken.c putobj.c

If the cfiles file contains more file names than fit on a single shell command line (up to LINE_MAX), the xargs command runs the lintcommand with the file names that fit. It then constructs and runs another lint command using the remaining file names. Depending on the names listed in the cfiles file, the commands might look like the following:

lint -a main.c readit.c gettoken.c . . .
lint -a getisx.c getprp.c getpid.c . . .
lint -a fltadd.c fltmult.c fltdiv.c . . .

This command sequence is not quite the same as running the lint command once with all the file names. The lint command checks cross-references between files. However, in this example, it cannot check between the main.c and the fltadd.c files, or between any two files listed on separate command lines.

  1. How to find a file older than X days

find .  -type f -mtime X  -print  (mtime: modified time)

  1. How to find file not accessed for X days

find . -type f -atime X -print

  1. How to use “OR” operator in find command

find . -atime +5 \ ( -name “*.o” -o -name “*.tmp” \ ) -print

  1. Time operators which can be used with “find” command

atime = access time (tells last access time)

mtime = access time (tells last modification time)

ctime = access time (tells last inode changes time)

  1. How to use find for exact  File time comparisons using find command

create files with exact times you would like to compare
touch -t 201109092100 file1

touch -t 201110102200 file 2

find . -newer file 1 \! -newer file 2 -print
This will list all the files created between these 2 dates.
-exec operator in “find” command
Often, when you find a file, you don’t just want to see its name; you want to do something, like grep for a text string. To do this, use the -exec operator. This allows you to specify a command that is executed upon each file that is found.
The -exec operator allows you to execute any command, including another find command. If you consider that for a moment, you realize that find needs some way to distinguish the command it’s executing from its own arguments. The obvious choice is to use the same end-of-command character as the shell (the semicolon). But since the shell uses the semicolon itself, it is necessary to escape the character with a backslash or quotes.
Therefore, every -exec operator ends with the characters \; There is one more special argument that find treats differently: {}. These two characters are used as the variable whose name is the file find found.
The difference between -exec and xargs is subtle. The first one will execute the program once per file, while xargs can handle several files with each process. However, xargs may have problems with filenames that contain embedded spaces

  1. how to delete strange file using -exec operator in “find”

find the inode number of the file using ls -il
find . -inum 163866 -exec rm {} \;

  1. how to search for all files in a folder with required word using “find” command

find . -name \*.log  -exec grep -n “root.conf” {} \;
grep can also search for the word “root.conf” but it’ll search in all files, not just only in *.log files

  1.  how to find all symbolic links in your home dir & prints the files to which your symbolic links point

find $HOME -type l -ls | awk ‘{print $NF}’

  1. To search file by size

to search file > 500Bytes  c=Size in bytes
find . -size 500c -print
to search files < 500 Bytes
find . -size -500c -print

  1. To search files by permission, user , group

find . -perm 755 -print
find . -user maq -print
find . -group maq  755 -print

  1. duplicating a directory tree using find command

find . -type d exec mkdir \usr\project\{} \;
using sed :
find . -type d -print | sed ‘s@^@/usr/project/@’ | xargs mkdir

find . -type d -print | sed ‘s@^@mkdir @’ | (cd /usr/project; sh)

  1. How to limit find to search only files at first level i.e. stop it from searching recusively

find . -maxdepth 1 -print

  1. How to limit find to search only files in one disk partition only

find /usr -size +10485760c -xdev -print

  1. fastest way to search files on linux

locate filename

  1. To search a particular word or string in all files in a directory & it’s subdirectory

egrep ‘string’ `find . -type f -print`
-bash-3.2$ egrep ‘string’ `find . -type f -print`

-bash: /bin/egrep: Argument list too long
Using the backquotes (“) might not work. If find finds too many files, egrep‘s command-line arguments can get too long. Using xargs can solve that; it splits long sets of arguments into smaller chunks
instead use

find . -type f -print -exec egrep ‘string’ \;
In case there’s only 1 file in the directory, where you are searching, the find will only print the string and not the file name, to resolve this
find . -type f -print -exec egrep ‘string’  /dev/null

  1. How to find the top 5 big files

find . -type f -exec ls -s {} \; | sort -n -r | head -5

  1. how to find all hidden files

find . -type f -name “.*”

  1. Using {} more than 1 time in find command

find . -name “*.txt” -exec cp {} {}.bkup \;

  1. Redirecting errors to /dev/null

find / -name “*.conf” 2>>/dev/null

  1. how to subsitute space with _ in file names

find . -type f -iname “*.txt″ -exec rename “s/ /_/g” {} \;
Linking, Renaming & copying files

  1. Directory is also just another file, to see it’s raw form

od -c
ls -if

  1. Hard link & Soft Link

Hard links

With a hard link, two filenames (i.e., two directory entries) point to the same inode and the same set of data blocks. All Unix versions support hard links. They have two important limitations: a hard link can’t cross a filesystem (i.e., both filenames must be in the same filesystem), and you can’t create a hard link to a directory (i.e., a directory can only have one name).[1] They have two important advantages: the link and the original file are absolutely and always identical, and the extra link takes no disk space (except an occasional extra disk block in the directory file).
Symbolic links (also called soft links or symlinks)

With a symbolic link, there really are two different files. One file contains the actual data; the other file just contains the name of the first file and serves as a “pointer.” We call the pointer the link. The system knows that whenever it opens a symlink, it should read the contents of the link and then access the file that really holds the data you want. Nearly all Unix systems support symbolic links these days. Symbolic links are infinitely more flexible than hard links. They can cross filesystems or even computer systems (if you are using NFS or RFS (Section 44.9)). You can make a symbolic link to a directory. A symbolic link has its own inode and takes a small amount of disk space to store.

  1. To Create a symllink/softlink

ln -s filename linkname

  1. To create a hardlink

ln filename linkname

Customizing your environment
What happens @ login:
First, login checks to see if you are not the root user and whether the file /etc/nologin exists, Second, login checks to see if there are any special conditions on your login attempt (which may be defined in /etc/usertty or /etc/securetty), check /etc/fbtab and may restrict your access to any devices listed in that file. These systems may also log failed login attempts to a file, such as /var/log/failedlogin, if it exists. login may also record the time of login in the file /var/log/lastlog, make an entry in the file /var/run/utmp, showing that you are successfully logged in (it is removed once you log out), and append the utmp entry to the file /var/log/wtmp , showing that you logged in. This wtmp record will be updated on logout, showing the duration of your login session.

  1. To check all shells on your box

cat /etc/shells

  1. To check your default shell

echo $SHELL

cat /etc/passwd

  1. Invoked as Interactive login shell: A login shell that you get after providing your username/password

file read:

/etc/profile

~/.bash_profile ; ~/.bash_profile; ~/.profile : first existing file is read

~/.bash_logout : On logout

  1. Invoked as Interactive Non-login Shell: You need not authenticate. For e.g.: When you open terminal from xterm icon

files read:

~/.bashrc
5.Invoked non interactively: All shell scripts use non interactive shells. They perform only the task mentioned in the script
files read:

BASH_ENV

  1. Invoked with the sh command

Bash behaves like sh and confirms to POSIX
files read:

/etc/profile

~/.profile

  1. posix mode

set -o posix

  1. invoked remotely

files read:

~/etc/bashrc

  1. Invoked when UID!= EUID

No startup file is read

  1. how to test if a shell is interactive shell

echo $-

i – interactive

  1. Differences in interactive mode:

Job control enabled by default.

Prompts are set, PS2 is enabled for multi−line commands, it is usually set to “>”. This is also the

prompt you get when the shell thinks you entered an unfinished command, for instance when you

forget quotes, command structures that cannot be left out, etc.

Commands are by default read from the command line using readline.

Bash interprets the shell option ignoreeof instead of exiting immediately upon receiving EOF

(End Of File).

Command history and history expansion are enabled by default. History is saved in the file pointed to

by HISTFILE when the shell exits. By default, HISTFILE points to ~/.bash_history.

In the absence of traps, the SIGTERM signal is ignored.

In the absence of traps, SIGINT is caught and handled. Thus, typing Ctrl+C, for example, will not

quit your interactive shell.

Sending SIGHUP signals to all jobs on exit is configured with the huponexit option.

  1. To view the directory stack

echo $DIRSTACK

dirs

  1. Bash supports 3 built ins: Bourne built-in; Bash built-in; Posix built-in
  1. To check exit status of last command: $?
  1. To run the script in debug mode

sh -x abc.sh
sh +x abc.sh

  1. To run script in other shells

bash -x abc.sh

ksh -x abc.sh

tcsh -x abc.sh
The specified shell will start as a subshell of your current shell and execute the script. This is done when you want the script to start up with specific options or under specific conditions which are not specified in the script.

  1. To execute the script in same shell

./abc.sh

source abc.sh

  1. Character Definition

$*

Expands to the positional parameters, starting from one. When the expansion occurs

within double quotes, it expands to a single word with the value of each parameter

separated by the first character of the IFS special variable.

$@

Expands to the positional parameters, starting from one. When the expansion occurs

within double quotes, each parameter expands to a separate word.

$# Expands to the number of positional parameters in decimal.

$? Expands to the exit status of the most recently executed foreground pipeline.

$−

A hyphen expands to the current option flags as specified upon invocation, by the set

built−in command, or those set by the shell itself (such as the −i).

$$ Expands to the process ID of the shell.

$!

Expands to the process ID of the most recently executed background (asynchronous)

command.

$0 Expands to the name of the shell or shell script.

$_

The underscore variable is set at shell startup and contains the absolute file name of the

shell or script being executed as passed in the argument list. Subsequently, it expands to

the last argument to the previous command, after expansion. It is also set to the full

pathname of each command executed and placed in the environment exported to that

command. When checking mail, this parameter holds the name of the mail file.

  1. Escape character: Escape characters are used to remove the special meaning from a single character. A non−quoted backslash, \,is used as an escape character in Bash.
  1. Single quotes (”) are used to preserve the literal value of each character enclosed within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash
  1. To create a variable on the fly and display assigned value

echo ${FRANKY:=franky}
Regular expressions:

A regular expression is a pattern that describes a set of strings. Regular expressions are constructed

analogously to arithmetic expressions by using various operators to combine smaller expressions
Operator Effect

. Matches any single character.

? The preceding item is optional and will be matched, at most, once.

* The preceding item will be matched zero or more times.

+ The preceding item will be matched one or more times.

{N} The preceding item is matched exactly N times.

{N,} The preceding item is matched N or more times.

{N,M} The preceding item is matched at least N times, but not more than M times.

− represents the range if it’s not first or last in a list or the ending point of a range in a list.

^ Matches the empty string at the beginning of a line; also represents the characters not in the

range of a list.

$ Matches the empty string at the end of a line.

\b Matches the empty string at the edge of a word.

\B Matches the empty string provided it’s not at the edge of a word.

\< Match the empty string at the beginning of word.

\> Match the empty string at the end of word.

  1. Word anchors: To search a pattern at the start of line : ^

To search a pattern @ end of line: $

  1. To search line starting with string “abcd”

grep ‘\<abcd\>’ server.txt

  1. To find a string that is a separate word (enclosed by spaces), it is better use the −w

for example to search “/” fs in /etc/fstab

if you give grep / /etc/fstab .. it’ll show all file systems having /

if you give grep -w /  /etc/fstab.. it’ll show only /root file system

  1. To search for . or * in a file

grep -F ‘*’ abc.sh

  1. To search all directory starting with a,b,c or x,y,z

ls -ld [a-cx-z]*

  1. To search all directory except the directories starting with a,b,c or x,y,z

ls -ld [^a-cx-z]*

  1. character classes

Character classes can be specified within the square braces, using the syntax [:CLASS:], where CLASS is defined in the POSIX standard and has one of the values “alnum”, “alpha”, “ascii”, “blank”, “cntrl”, “digit”, “graph”, “lower”, “print”, “punct”, “space”, “upper”, “word” or “xdigit”.
SED

A Stream EDitor is used to perform basic transformations on text read from a file or a pipe. The result is sent to standard output. The syntax for the sed command has no output file specification, but results can be saved to a file using output redirection. The editor does not modify the original input.
What distinguishes sed from other editors, such as vi and ed, is its ability to filter text that it gets from a

pipeline feed. You do not need to interact with the editor while it is running; that is why sed is sometimes

called a batch editor. This feature allows use of editing commands in scripts, greatly easing repetitive editing tasks. When facing replacement of text in a large number of files, sed is a great help.
Command Result

a\ Append text below current line.

c\ Change text in the current line with new text.

d Delete text.

i\ Insert text above current line.

p Print text.

r Read a file.

s Search and replace text.

w Write to a file.
Option Effect

−e SCRIPT

Add the commands in SCRIPT to the set of commands to be run while processing the

input.

−f

Add the commands contained in the file SCRIPT−FILE to the set of commands to be

run while processing the input.

−n Silent mode.

−V Print version information and exit.

  1. To print lines containing a pattern

sed ‘/string/p’ textfile

  1. To print only the lines matching the pattern

sed -n ‘/string/p’ textfile

  1. To delete the line containing the pattern

sed -n ‘/string/d’ textfile

  1. Find and replace with Sed

sed ‘s/errors/awesome’ abc.log

Note: This will change only 1st occurence of “errors” in the line

  1. Find and replace with Sed globally

sed ‘s/errors/awesome/g’ abc.log

  1. To insert a string “>” at the beginning of each line of a file

sed ‘s/^/> /’ abc.log

  1. To insert a string “EOL” at the end of each line of a file

sed ‘s/$/EOL/g’ abc.log

  1. To run multiple replace commands

sed ‘s/^/>/g’ -e ‘s/$/EOL/g’ abc.log
Keep in mind that by default sed prints its results to the standard output, most likely your terminal window. If you want to save the output to a file, redirect it

sed option ‘some/expression’ file_to_process > sed_output_in_a_file
AWK

The basic function of awk is to search files for lines or other text units containing one or more patterns. When a line matches one of the patterns, special actions are performed on that line.Programs in awk are different from programs in most other languages, because awk programs are “data−driven”: you describe the data you want to work with and then what to do when you find it
There are several ways to run awk. If the program is short, it is easiest to run it on the command line:

awk PROGRAM inputfile(s)

If multiple changes have to be made, possibly regularly and on multiple files, it is easier to put the awk

commands in a script. This is read like this:

awk −f PROGRAM−FILE inputfile(s)

How to print/display the first line of a file?

head -1 file.txt

sed ‘2,$ d’ file.txt

How to print/display the last line of a file?

tail -1 file.txt

sed -n ‘$ p’ test

How to display n-th line of a file?

sed –n ‘<n> p’ file.txt

Disclaimer:

All the trademarked and copyrighted content in my blog belong to their original owners and are included for representation purposes only.

Short script to remove docker images

  1. Create a file: remdockerimages.sh on your docker host machine
  2. Copy the below content in remdockerimages.sh
  3. #!/bin/bash
    rm imagesInfo.txt
    sudo docker images | awk ‘{print $3}’ > imagesInfo.txt
    echo “$(tail -n +2 imagesInfo.txt)” > imagesInfo.txt
    while read line; do
    sudo docker rmi “${line}”
    done < imagesInfo.txt
  4. Save the file and run ./remdockerimages.sh imagesInfo.txt
  5. Enjoy 🙂

GITHUB: https://github.com/tryinggithubnow/smalltools/blob/master/remdockerimages.sh

Short Script to remove all stopped/exited docker containers

  1. Create a file: remdockerinstances.sh on your docker host machine
  2. Copy the below content in remdockerinstances.sh
  3. #!/bin/bash
    rm instanceinfo.txt
    sudo docker ps -a | awk ‘{print $1}’ > instanceinfo.txt
    echo “$(tail -n +2 instanceinfo.txt)” > instanceinfo.txt
    while read line; do
    sudo docker rm “${line}”
    done < instanceinfo.txt
  4. Save the file and run ./remdockerinstances.sh instanceinfo.txt
  5. Enjoy 🙂

Docker Goodies

Thanks to Docker 🙂

Self-Paced Training for Docker

https://training.docker.com/self-paced-training

Docker Blogs

http://blog.docker.com (http://blog.docker.com/)

Docker Documents

http://docs.docker.com (http://docs.docker.com/)

References

https://github.com/kencochrane/docker-guidebook/blob/master/docker-guidebook.rst

https://github.com/wsargent/docker-cheat-sheet

http://www.oracle.com/technetwork/middleware/weblogic/overview/weblogic-server-docker-containers-2491959.pdf

http://linuxconfig.org/docker-container-backup-and-recovery

http://developerblog.redhat.com/2014/05/15/practical-introduction-to-docker-containers/

http://www.openstack.org/containers?gclid=CIDah9XDoMgCFVcVjgodkBADJw

http://blog.flux7.com/blogs/docker/docker-commands

Disclaimer:

All the trademarked and copyrighted content in my blog belong to their original owners and are included for representation purposes only.

A fresh start

Hi,

I’m starting this blog dedicated to DevOps and more. I’ll try to keep this site updated with the DevOps related activities that I do, posts, articles, documents, the issues which i face while working with various DevOps tools & their Solutions, suggestions etc.

Live and Let Live DevOps

MAQ