[NOVA]Complete Installation of Multi Servers ---in detail

Multi Servers installation  

ubuntu

HOSTS	NIC	Networks	IP	gateway
ubuntu1	eth0	172.16.2.0/24	172.16.2.79	172.16.2.1
ubuntu1	eth1	192.168.1.0/24	192.168.1.1	192.168.1.2
ubuntu2	eth0	172.16.2.0/24	172.16.2.78	172.16.2.1
ubuntu2	eth1	192.168.1.0/24	192.168.1.2	none
ubuntu3~6	eth1	192.168.1.0/24	192.168.1.3	192.168.1.2

Match to this topology

http://hugokuo-hugo.blogspot.com/2011/04/this-topology-works-fine-with-bzr-984.html

HOST
ubuntu1 nova-api nova-objectstore mysql RabbitMQ
ubuntu2 nova-network nova-scheduler
ubuntu3 nova-compute

Add Nova repository on Ubuntu , We can use Release-Version or Latest-Trunk by replace "release" to "trunk" 

sudo apt-get update
sudo apt-get install python-software-properties -y
sudo add-apt-repository ppa:nova-core/trunk
sudo apt-get update

Packages
Ubuntu1 nova-api nova-objectstore mysql-server RabbitMQ phpmyadmin

Ubuntu2 nova-network nova-scheduler *dnsmasq *bridge-utils

ubuntu3 nova-compute dnamasq bridge-utils

Install Packages

Ubuntu1

Install RabbitMQ

apt-get install -y rabbitmq-server

Nova API & Objectstore services

apt-get install nova-api nova-objectstore

The following extra packages will be installed:
  libjs-jquery libmysqlclient16 mysql-common nova-common python-amqplib
  python-anyjson python-argparse python-boto python-carrot python-cheetah
  python-cjson python-daemon python-decorator python-dingus python-eventlet
  python-formencode python-gflags python-glance python-greenlet python-httplib2
  python-ipy python-ldap python-libxml2 python-lockfile python-m2crypto
  python-migrate python-mysqldb python-netaddr python-nose python-nova
  python-novaclient python-openid python-openssl python-pam python-paste
  python-pastedeploy python-pastescript python-pkg-resources python-prettytable
  python-pycurl python-routes python-scgi python-serial python-setuptools
  python-sqlalchemy python-sqlalchemy-ext python-stompy python-tempita
  python-twisted-bin python-twisted-core python-twisted-web python-webob
  python-zope.interface
Suggested packages:
  javascript-common python-argparse-doc python-cjson-dbg python-egenix-mxdatetime
  python-dns python-greenlet-doc python-greenlet-dev python-greenlet-dbg
  python-ldap-doc mysql-server-5.1 mysql-server python-mysqldb-dbg ipython
  python-coverage python-suds python-openssl-doc python-openssl-dbg python-pam-dbg
  python-pastewebkit libapache2-mod-wsgi libapache2-mod-python libapache2-mod-scgi
  python-pgsql libjs-mochikit python-cherrypy3 python-cherrypy python-flup
  python-distribute python-distribute-doc libcurl4-gnutls-dev python-pycurl-dbg
  python-wxgtk2.8 python-wxgtk2.6 python-wxgtk python-sqlalchemy-doc
  python-psycopg2 python-kinterbasdb python-pymssql python-twisted-bin-dbg
  python-tk python-gtk2 python-glade2 python-qt3 python-profiler
  python-z3c.recipe.sphinxdoc
The following NEW packages will be installed:
  libjs-jquery libmysqlclient16 mysql-common nova-api nova-common nova-objectstore
  python-amqplib python-anyjson python-argparse python-boto python-carrot
  python-cheetah python-cjson python-daemon python-decorator python-dingus
  python-eventlet python-formencode python-gflags python-glance python-greenlet
  python-httplib2 python-ipy python-ldap python-libxml2 python-lockfile
  python-m2crypto python-migrate python-mysqldb python-netaddr python-nose
  python-nova python-novaclient python-openid python-openssl python-pam
  python-paste python-pastedeploy python-pastescript python-pkg-resources
  python-prettytable python-pycurl python-routes python-scgi python-serial
  python-setuptools python-sqlalchemy python-sqlalchemy-ext python-stompy
  python-tempita python-twisted-bin python-twisted-core python-twisted-web
  python-webob python-zope.interface

Actually , I think python-suds should be installed too.

Install MySQL , plz remember the password which will set in nova.conf later.

sudo apt-get install mysql-server
sed -i 's/127.0.0.1/0.0.0.0/g' /etc/mysql/my.cnf
service mysql restart

Install PHPmyadmin

sudo apt-get install phpmyadmin

Install other tools

sudo apt-get install euca2ools unzip

Ubuntu2

sudo apt-get install nova-network nova-scheduler dnsmasq bridge-utils

The following extra packages will be installed:
  libmysqlclient16 libreadline5 mysql-common nova-common python-amqplib python-anyjson python-boto python-carrot python-cheetah python-daemon python-decorator python-dingus
  python-eventlet python-formencode python-gflags python-glance python-greenlet python-ipy python-ldap python-lockfile python-m2crypto python-migrate python-mysqldb
  python-netaddr python-nose python-nova python-novaclient python-openid python-paste python-pastedeploy python-pastescript python-prettytable python-routes python-scgi
  python-setuptools python-sqlalchemy python-sqlalchemy-ext python-stompy python-tempita python-webob socat vlan
Suggested packages:
  resolvconf radvd python-dns python-greenlet-doc python-greenlet-dev python-greenlet-dbg python-ldap-doc mysql-server-5.1 mysql-server python-mysqldb-dbg ipython
  python-coverage python-suds python-pastewebkit libapache2-mod-wsgi libapache2-mod-python libapache2-mod-scgi python-pgsql libjs-mochikit python-cherrypy3 python-cherrypy
  python-flup python-sqlalchemy-doc python-psycopg2 python-kinterbasdb python-pymssql
The following NEW packages will be installed:
  bridge-utils dnsmasq libmysqlclient16 libreadline5 mysql-common nova-common nova-network nova-scheduler python-amqplib python-anyjson python-boto python-carrot
  python-cheetah python-daemon python-decorator python-dingus python-eventlet python-formencode python-gflags python-glance python-greenlet python-ipy python-ldap
  python-lockfile python-m2crypto python-migrate python-mysqldb python-netaddr python-nose python-nova python-novaclient python-openid python-paste python-pastedeploy
  python-pastescript python-prettytable python-routes python-scgi python-setuptools python-sqlalchemy python-sqlalchemy-ext python-stompy python-tempita python-webob socat
  vlan
0 upgraded, 46 newly installed, 0 to remove and 289 not upgraded.
Need to get 8,008kB of archives.

Install other tools

sudo apt-get install unzip

Ubuntu3

sudo apt-get install nova-compute dnsmasq bridge-utils python-suds

Reading state information... Done
The following extra packages will be installed:
  ajaxterm apache2 apache2-mpm-worker apache2-utils apache2.2-bin apache2.2-common cpu-checker curl dnsmasq-base ebtables gawk kpartx kvm libaio1 libapr1 libaprutil1
  libaprutil1-dbd-sqlite3 libaprutil1-ldap libasound2 libavahi-client3 libavahi-common-data libavahi-common3 libcurl3 libflac8 libice6 libjs-jquery libmysqlclient16 libnl1
  libogg0 libpciaccess0 libpulse0 libpython2.6 libsdl1.2debian libsdl1.2debian-alsa libsm6 libsndfile1 libvirt-bin libvirt0 libvorbis0a libvorbisenc2 libx11-xcb1 libxcb-atom1
  libxen3 libxi6 libxml2-utils libxtst6 mysql-common nova-common open-iscsi open-iscsi-utils python-amqplib python-anyjson python-argparse python-boto python-carrot
  python-cheetah python-cjson python-daemon python-decorator python-dingus python-eventlet python-formencode python-gflags python-glance python-greenlet python-httplib2
  python-ipy python-ldap python-libvirt python-libxml2 python-lockfile python-m2crypto python-migrate python-mysqldb python-netaddr python-nose python-nova python-novaclient
  python-openid python-openssl python-pam python-paste python-pastedeploy python-pastescript python-pkg-resources python-prettytable python-pycurl python-routes python-scgi
  python-serial python-setuptools python-sqlalchemy python-sqlalchemy-ext python-stompy python-tempita python-twisted-bin python-twisted-core python-twisted-web python-webob
  python-zope.interface qemu-common qemu-kvm seabios ssl-cert vgabios x11-common
Suggested packages:
  apache2-doc apache2-suexec apache2-suexec-custom resolvconf libasound2-plugins javascript-common pulseaudio policykit-1 lvm2 python-argparse-doc python-cjson-dbg
  python-egenix-mxdatetime python-dns python-greenlet-doc python-greenlet-dev python-greenlet-dbg python-ldap-doc mysql-server-5.1 mysql-server python-mysqldb-dbg ipython
  python-coverage python-openssl-doc python-openssl-dbg python-pam-dbg python-pastewebkit libapache2-mod-wsgi libapache2-mod-python libapache2-mod-scgi python-pgsql
  libjs-mochikit python-cherrypy3 python-cherrypy python-flup python-distribute python-distribute-doc libcurl4-gnutls-dev python-pycurl-dbg python-wxgtk2.8 python-wxgtk2.6
  python-wxgtk python-sqlalchemy-doc python-psycopg2 python-kinterbasdb python-pymssql python-twisted-bin-dbg python-tk python-gtk2 python-glade2 python-qt3 python-profiler
  python-z3c.recipe.sphinxdoc kvm-pxe mol-drivers-macosx openbios-sparc ubuntu-vm-builder uml-utilities openssl-blacklist
Recommended packages:
  python-psyco
The following NEW packages will be installed:
  ajaxterm apache2 apache2-mpm-worker apache2-utils apache2.2-bin apache2.2-common bridge-utils cpu-checker curl dnsmasq dnsmasq-base ebtables gawk kpartx kvm libaio1 libapr1
  libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libasound2 libavahi-client3 libavahi-common-data libavahi-common3 libcurl3 libflac8 libice6 libjs-jquery
  libmysqlclient16 libnl1 libogg0 libpciaccess0 libpulse0 libpython2.6 libsdl1.2debian libsdl1.2debian-alsa libsm6 libsndfile1 libvirt-bin libvirt0 libvorbis0a libvorbisenc2
  libx11-xcb1 libxcb-atom1 libxen3 libxi6 libxml2-utils libxtst6 mysql-common nova-common nova-compute open-iscsi open-iscsi-utils python-amqplib python-anyjson
  python-argparse python-boto python-carrot python-cheetah python-cjson python-daemon python-decorator python-dingus python-eventlet python-formencode python-gflags
  python-glance python-greenlet python-httplib2 python-ipy python-ldap python-libvirt python-libxml2 python-lockfile python-m2crypto python-migrate python-mysqldb
  python-netaddr python-nose python-nova python-novaclient python-openid python-openssl python-pam python-paste python-pastedeploy python-pastescript python-pkg-resources
  python-prettytable python-pycurl python-routes python-scgi python-serial python-setuptools python-sqlalchemy python-sqlalchemy-ext python-stompy python-suds python-tempita
  python-twisted-bin python-twisted-core python-twisted-web python-webob python-zope.interface qemu-common qemu-kvm seabios ssl-cert vgabios x11-common
0 upgraded, 110 newly installed, 0 to remove and 41 not upgraded.
Need to get 24.4MB of archives.
After this operation, 99.4MB of additional disk space will be used.

Install other tools

sudo apt-get install unzip

Configuration

The most important stuff is Network mode
There are three modes support in Nova , depends on your environment.
*Flat
*FlatDHCP
*Vlan

/etc/nova/nova.conf

--sql_connection=mysql://root:nova@<$mysql_server_IP>/nova
--s3_host=<$Objectstore_IP>
--rabbit_host=<$RabbitMQ_IP>
--cc_host=<API_Server_IP>
--ec2_url=http://<$API_Server_IP>:8773/services/Cloud
--daemonize=1
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--FAKE_subdomain=ec2
--ca_path=/var/lib/nova/CA
--keys_path=/var/lib/nova/keys
--networks_path=/var/lib/nova/networks
--instances_path=/var/lib/nova/instances
--images_path=/var/lib/nova/images
--buckets_path=/var/lib/nova/buckets
--libvirt_type=<$type_of_Hypervisor>
--network_manager=nova.network.manager.<$Network_mode> # FlatManager / FlatDHCPManager / VlanManager
--flat_interface=<$flat_network_NIC>                   # What if there's only one NIC?
--logdir=/var/log/nova
--verbose
--fixed_range=<All_project_Network>
--network_size=<Number of IPs in this Network>

An Example as below

--sql_connection=mysql://root:nova@192.168.1.1/nova
--s3_host=192.168.1.1
--rabbit_host=192.168.1.1
--cc_host=192.168.1.1
--ec2_url=http://192.168.1.1:8773/services/Cloud
--daemonize=1
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--FAKE_subdomain=ec2
--ca_path=/var/lib/nova/CA
--keys_path=/var/lib/nova/keys
--networks_path=/var/lib/nova/networks
--instances_path=/var/lib/nova/instances
--images_path=/var/lib/nova/images
--buckets_path=/var/lib/nova/buckets
--libvirt_type=kvm
--network_manager=nova.network.manager.FlatManager
--logdir=/var/log/nova
--verbose
--fixed_range=192.168.1.0/24
--network_size=256

Copy nova.conf to all service hosts...
scp nova.conf ubuntu1:/ubuntu2:/ubunt3:

Configure MySQL DB on ubuntu1

Create nova database

mysql -uroot -p$MYSQL_PASS -e 'CREATE DATABASE nova;'

Update the DB to include user 'root'@'%' with super user privileges:

mysql -uroot -p$MYSQL_PASS -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;" 

Set mySQL root password:

mysql -uroot -p$MYSQL_PASS -e "SET PASSWORD FOR 'root'@'%' = PASSWORD('$MYSQL_PASS');" 
service mysql restart

SETUP NOVA ENVIRONMENT (with root user)

on Ubuntu1

nova-manage db sync
nova-manage user admin <user_name>
nova-manage project create <project_name> <user_name>
nova-manage network create <project-network> <number-of-networks-in-project> <IPs in project>

Create Nova certifications

nova-manage project zipfile $NOVA_PROJECT $NOVA_PROJECT_USER <$PATH>/novacreds.zip

ex.
nova-manage project zipfile hugopro hugo ~/novacreds.zip

unzip novacreds.zip
source ~/novarc

Source it when bootup

cat ~/novarc >> ~/.bashrc

After all components is ready , restart all service

service libvirt-bin restart; service nova-network restart; service nova-compute restart; service nova-api restart; service nova-objectstore restart; service nova-scheduler restart;

On ubuntu1

service nova-api restart;service nova-objectstore restart;

On ubuntu2

service nova-network restart;service nova-scheduler restart;

On ubuntu3

service libvirt-bin restart;service nova-compute restart;

Closing steps, and cleaning up

Allow ping & SSH to instance

euca-authorize -P icmp -t -1:-1 default
euca-authorize -P tcp -p 22 default

Download image or from http://uec-images.ubuntu.com/

image="ttylinux-uec-amd64-12.1_2.6.35-22_1.tar.gz" 
wget http://smoser.brickies.net/ubuntu/ttylinux-uec/$image
uec-publish-tarball $image mybucket

Add keypair

euca-add-keypair hugokey > hugokey.priv
chmod 600 hugokey.priv

Troubleshooting

1. While nova-network service can not be ran up , if there's a error about dnsmasq?
A. 

killall dnsmasq
service nova-network restart

2. While the instance failed to spawn
A.

killall dnsmasq
service nova-compute restart

3. Don't forget to add ipv4 forward on NOVA-NETWORK

vi /etc/sysctl.conf   #turn on ipv4forward
sysctl -p /etc/sysctl.conf

try other image first

check those issue
https://answers.launchpad.net/nova/+question/151725
Using Glance service or NFS share image from fronend

https://answers.launchpad.net/nova/+question/151125
sudo modprobe kvm
sudo chgroup kvm /dev/kvm
sudo chmod g+w /dev/kvm

• iptable rule
• nova.conf
• ifconfig