Friday, February 26, 2016

Nova Docker Installation

After going through a lot of links (some good, some confusing), I finally figured out the installation of nova-docker. So whats the whole idea ?
When we create an OpenStack compute node with kvm as the underlying hypervisor, the VMs on that node get created using kvm hypervisor. Simple..
In the same way, there's a nova-docker driver. When we use nova-docker driver on a compute node, we can create docker containers using the "nova boot" command. That's what I did on my OpenStack kilo setup.

I created an All-in-one OpenStack kilo setup and used qemu as the hypervisor. Then, I thought of adding another compute node to the setup - but this one with nova-docker driver. Without any delay, I'll tell you how I did it (Though you would find a lot of links on it, but I prefer writing it my way :) ).

Step 1: Install docker on your VM - http://docs.docker.io/en/latest/installation/
Step 2: usermod -aG docker nova
Step 3: pip install docker-py
Step 4: git clone https://github.com/stackforge/nova-docker
[Steps 1-4 copied shamelessly from here]
Step 5: git checkout stable/kilo
Step 6: cd nova-docker
    Step 6.1: Edit the file driver.py - vi novadocker/virt/docker/driver.py as explained below
                   In def spawn(...), args{} needs to be edited as given below:

                    args = {
                   'hostname': instance['name'],
                   'mem_limit': self._get_memory_limit_bytes(instance),
                   'cpu_shares': self._get_cpu_shares(instance),
                   'network_disabled': True,
                   'tty': True,   # This line needs to be added.
        }
[Now again, step 6.1 is copied from here.]
Step 7: Follow the steps to configure and install a compute node from Openstack Kilo guide. - http://docs.openstack.org/kilo/install-guide/install/apt/content/neutron-compute-node.html
Step 8:python setup.py install - Install the nova-docker driver.
Step 9: Now we need to make some modifications to the nova configuration files that we edited in step 7. In "/etc/nova/nova.conf" under DEFAULT section, set 
compute_driver = novadocker.virt.docker.DockerDriver 
In /etc/nova/nova-compute.conf, we will set the compute driver as novadocker [that we installed in step 8].
cat /etc/nova/nova-compute.conf 
[DEFAULT]
#compute_driver=libvirt.LibvirtDriver
compute_driver=novadocker.virt.docker.DockerDriver
[libvirt]
virt_type=qemu
Step 10: Create the directory /etc/nova/rootwrap.d, if it does not already exist, and inside that directory create a file "docker.filters" with the following content:

# nova-rootwrap command filters for setting up network in the docker driver
# This file should be owned by (and only-writeable by) the root user

[Filters]
# nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*'
ln: CommandFilter, /bin/ln, root

Step 11: In "/etc/glance/glance-api.conf", set 
container_formats=ami,ari,aki,bare,ovf,ova,docker

Step 12: Follow "Uploading Images to Glance" onwards from this. If you get any error, follow this link. I got some of the same errors as described in this wonderful link. Many thanks to the author.

Once you are done with this, I would suggest you to create a new availability zone for this compute host. This way, while creating the docker instance, you can choose the docker availability zone. Here's what I did,

Step 1: nova aggregate-create docker-aggregate docker-availability-zone 
Step 2: nova aggregate-add-host

Now you can create an instance using the docker image. Check your instance using docker ps command on the compute node that we just created.

Thats all folks, I have consolidated everything in this blog post along with the helpful links. 

Putting all the links that I used for reference:

Sunday, June 14, 2015

Orchestration with Heat

This time, I tried to get my hands dirty by writing a heat script - my first yaml file.
Oops!!!
I think I need to start off with a small intro.
OK, so what is openstack heat ?
Heat is one of the several services in Openstack. It is used for orchestration. This simply means that creating networks, instances, routers, security groups etc using a script(or rather a template in YAML format - called HOT - Heat Orchestration Template) instead of browsing the horizon or using nova/neutron etc commands to setup a tenant. It also means that if the same needs to be replicated, you just need to provide some parameters and its done. And if you want to delete the same, just do it by hitting the enter key with heat stack's delete command.
Ok, I used another jargon here... Heat stack. Heat stack can be simply defined as the collection of resources that are orchestrated with heat. So if you have spawned 5 networks, 100 VMs, 10 routers, a few security group rules through a heat yaml, then all of these form a heat stack.


In this post, I would describe a HOW-TO for writing a heat template. I have created a simple YAML file to spawn 2 VMs, a security group and a network to which these VMs are attached. Once a VM is spawned, it executes a script defined in the user_data section. This script prints "Scripts loaded on startup" in syslog. The purpose of this script is to illustrate that users can create VMs and spawn a configuration script or a start-up script once the VM boots up.

Sections:
A heat template has the below sections:
1. Heat template version
2. Parameters
3. Resources
4. Outputs - (Not describing in this post.)

Heat template version is simply a header that describes the template version that you

The parameters section is used to define the parameters. These parameters are like input variables with some default values.
For example, if you want to create two stacks for two different tenants using the below yaml file, then you can use the parameters to specify the values like vm name, subnet address etc for the tenant.

The resources section describes the resources of your stack - In this section, you define the VMs, networks, security groups, ports etc.

heat_template_version: 2013-05-23
description: Create a network and attach a VM. Run a script in the VM

parameters:
  p_my_vm_flavor:
    type: string
    label: Flavor
    description:
    default:ubuntu

  p_my_private_network:
    type: string
    description:
    default: tenantA_net

  p_my_private_subnet:
    type: string
    description:
    default: tenantA_subnet

  p_my_network_cidr:
    type: string
    description:
    default: 11.0.0.0/24

  p_my_network_gateway:
    type: string
    description:
    default: 11.0.0.1

resources:
  r_my_network:
    type: OS::Neutron::Net
    properties:
      name: { get_param: p_my_private_network }
  r_my_network_subnet:
    type: OS::Neutron::Subnet
    properties:
      name: {get_param: p_my_private_subnet }
      network_id: {get_resource: r_my_network}
      gateway_ip: {get_param: p_my_network_gateway}
      cidr: { get_param: p_my_network_cidr }
  r_my_port:
    type: OS::Neutron::Port
    properties:
      network_id: {get_resource: r_my_network}
  r_security_group:
    type: OS::Neutron::SecurityGroup
    properties:
      description:
      name:
      rules: [ { "direction": ingress, "protocol": ICMP }, { "direction": ingress, "protocol": TCP,"port_range_min": 1, "port_range_max": 65535 }, { "direction": ingress, "protocol": UDP,  "port_range_min": 1, "port_range_max": 65535 } ]

  r_my_vm:
    type: OS::Nova::Server
    properties:
      image: firewall
      flavor: m1.medium
      networks:
        - port: {get_resource: r_my_port}
      user_data: |
        #!/bin/sh -ex
        logger "Script loaded on startup"
  r_my_vm_port:
    type: OS::Neutron::Port
    properties:
      network_id: {get_resource: r_my_network}
      security_groups: [{get_resource: r_security_group}]


Parameters section: I have used 5 configurable parameters and specified  a default value (default: tag) for each of this. So if a user doesn't provides a parameter, its default value would be used.
  p_my_vm_flavor - To indicate the flavor of VM.
  p_my_private_network - Specify name of tenant's network.
  p_my_private_subnet - Specify the private subnet of the tenant.
  p_my_network_cidr - CIDR of tenant's network.
  p_my_network_gateway - Gateway of the tenant's network.

Resources Section: I have used 6 type of resources
r_my_network: It is defined as OS::Neutron::Net. This is an OpenStack resource type (see this).
Similarly  I have defined r_my_network_subnet, r_my_port, r_security_group, r_my_vm and r_my_vm_port. All these are self explanatory and define the subnet, port, security group, VM and VM's port.

The user_data in r_my_vm properties defines a script that would be loaded on start when the VM is instantiated.

You can run this YAML file(test.aml) with "heat stack-create" command and provide the below file as parameters.

parameters:
  p_my_private_network: TenantCNetwork
  p_my_private_subnet: TenantCSubnet
  p_my_network_cidr: 80.0.0.1/24
  p_my_network_gateway: 80.0.0.1


Friday, May 8, 2015

NORMAL mode in OVS - MAC learning

OpenVSwitch or OVS can run in two modes
1. NORMAL mode
2. Flow mode

In NORMAL mode, OVS works like any other L2 layer switch operating on MAC-Port mapping.
To use an OVS in NORMAL mode, simple create an OVS bridge and add the below flow entry
ovs-vsctl add-br br0
ovs-vsctl add-port eth0
ovs-ofctl add-flow br0 action=NORMAL
For flows hitting this entry, the MAC table maintained by OVS would be referred for forwarding decision.

In Flow mode, the regular OpenFlow pipeline is hit and OVS works on the basis of flows installed.

In my experiment, I would explain how the NORMAL mode works standalone and also how it works in conjunction with flow mode.
I have only used mininet and OVS for this experiment on Ubuntu 14.04 LTS.

Step 1: Create a simple topology with one switch and three hosts.

$ sudo mn --topo=single,3 --controller=none --mac
*** Creating network
*** Adding controller
*** Adding hosts:
h1 h2 h3
*** Adding switches:
s1
*** Adding links:
(h1, s1) (h2, s1) (h3, s1)
*** Configuring hosts
h1 h2 h3
*** Starting controller
*** Starting 1 switches
s1
*** Starting CLI:

Step 2: Execute ovs-appctl command. This command displays the MAC table created by OVS. Since we have not done any ping, the table doesn't has the MAC entries for h1,h2 and h3.
mininet> sh ovs-appctl fdb/show s1
 port  VLAN  MAC                Age
LOCAL     0  b2:c9:fc:3c:1a:41    8
mininet>

Step 3: We do not have any flows installed on OVS. If you try to ping between hosts, nothing would happen.
mininet> sh ovs-ofctl dump-flows s1
NXST_FLOW reply (xid=0x4):
mininet>

Step 4: Now, lets add a NORMAL action flow entry on OVS and see what happens.
mininet> sh ovs-ofctl add-flow s1 action=normal
mininet> sh ovs-ofctl dump-flows s1
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=12.023s, table=0, n_packets=0, n_bytes=0, idle_age=12, actions=NORMAL
mininet>

The MAC table is still blank as in step 2.
mininet> sh ovs-appctl fdb/show s1
 port  VLAN  MAC                Age
LOCAL     0  b2:c9:fc:3c:1a:41   56
mininet>

Step 5: Lets ping
mininet> h1 ping h2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.587 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.085 ms
^C
--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.085/0.336/0.587/0.251 ms

The ping is successful because of NORMAL flow entry. The MAC addresses of the hosts h1 and h2 are learnt. h3 is still not learnt as there has been no ping to/from h3.

mininet> sh ovs-appctl fdb/show s1
 port  VLAN  MAC                Age
LOCAL     0  b2:c9:fc:3c:1a:41   63
    2     0  00:00:00:00:00:02    1
    1     0  00:00:00:00:00:01    1
mininet>

The above steps show the OVS operation in NORMAL mode.
Step 6 onwards, I would explain how NORMAL and FLOW mode work in conjunction,

Step 6: Now, lets add a flow for MAC address and port of h3 with a higher priority.
mininet> sh ovs-ofctl add-flow s1 priority=60000,dl_dst=00:00:00:00:00:03,actions=output:3
mininet> sh ovs-ofctl dump-flows s1
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=4.242s, table=0, n_packets=0, n_bytes=0, idle_age=4, priority=60000,dl_dst=00:00:00:00:00:03 actions=output:3
 cookie=0x0, duration=71.223s, table=0, n_packets=8, n_bytes=560, idle_age=43, actions=NORMAL
mininet>

Step 7: Lets ping and notice n_packets/n_bytes of flow entries.

mininet> h1 ping h3
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=0.558 ms
64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.090 ms
^C
--- 10.0.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.090/0.324/0.558/0.234 ms
mininet> sh ovs-ofctl dump-flows s1
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=13.124s, table=0, n_packets=2, n_bytes=196, idle_age=2, priority=60000,dl_dst=00:00:00:00:00:03 actions=output:3  --> Flow1
 cookie=0x0, duration=80.105s, table=0, n_packets=12, n_bytes=840, idle_age=2, actions=NORMAL ---> Flow2
mininet>

Notice that the packets/bytes increase for both the entries. This is because when h1 ping is sent to h3, Flow 1 is hit and for the ping reply, Flow 2 gets hit.

Journey of a packet when a VM accesses internet in Openstack

I am putting down my understanding in the post to explain the journey of a packet from a VM to an external network. As you read on, I would explain this figure in detail at interface and bridge level with the help of some slides.

Step 1: VM to br-int - Packet filtering by Security Group


Each VM that is created, is attached to a TAP interface (vnetX). This tap interface is connected to a Linux bridge qbrXXX and then a veth pair qbrXXXX - qvoXXXX connects the Linux bridge with br-int.
The security groups are implemented on TAP devices using iptables rules. If an instance has multiple ports, the same security groups are applied on all ports of the instances.

Step 2: br-int to br-tun (Inside the Compute node)


br-int and br-tun are connected via patch ports. The external packets (VLAN tagged) reach br-tun via the patch ports. On br-tun, the VLAN tag is stripped and a tunnel id is added to send the packet to the tunnel between the compute node and the network node.

Step 3: Packet travels to Network Node through GRE Tunnel
At this point, the packet reaches the physical interface - eth1 of the network node via a GRE tunnel.


Step 4: Packet reaches br-int from br-tun

eth1 of the network node belongs to br-tun. The packet is thus received by br-tun. br-tun removes the GRE header and sends the packet to br-int via patch ports(qr veth pair,i.e the receiving interface on br-int is qrXXXX). This is done via GRE-VLAN mapping maintained as flow rules on br-tun.

Step 5: Firewall rules on network node

The packet exits br-int via qrXXXX interface which exists in the qROUTER namespace that belongs to the tenant. Both qrXXXX and qgXXXX interfaces exist in the qROUTER namespace. You can check the interface and route and iptables details using the below commands.
#ip netns exec ifconfig -a
#ip netns exec route -n
#ip netns exec iptables -L
#ip netns exec iptables -L -t nat
qrXXXX is the interface that serves as the internal gateway for a tenant.
qgXXXX is the interface towards the external network on br-ex.
Rules of the tenant's firewall then get executed which determines whether the packet going to external network should be dropped or allowed.
NATing is also done at this point, so the packet leaving the network node has the source IP as that of the qROUTER's external gateway.
Once allowed, the packet reaches qgXXXX interfaces on br-ex and is set to external network or the internet.


The response takes the same path in reverse direction.

Saturday, March 21, 2015

OpenStack and SDN - The neutron buzz !!

Lately, I have been reading quite a lot about OpenStack and all the buzz around neutron and SDN. You would find many links and a plethora of information online. This post is an effort to highlight or describe in simple words, the role of OpenStack neutron in connection to SDN. So without wasting any time, lets start.

We all know that Neutron is the OpenStack's project to offer Network as a Serivce - NaaS. Neutron started as a separate project after it was separated from nova-network.
With the growing interest in OpenStack, a lot of companies are making efforts to integrate their SDN solutions with OpenStack.

How are they (companies) doing it ?



The above figure is the answer to what the companies are doing. The top layer is the application layer which calls the neutron service APIs(1). The neutron service APIs are the APIs which are exposed to OpenStack services, say horizon (For example, when you do some network configuration from the OpenStack GUI). 

Between the neutron service API and physical layer, lies the neutron plugin.
To write a neutron plugin, the vendor/company/individual should adhere with the below rule:
1. Implement the interface called by Neutron service APIs(3).
Thus, at any point, interface 1 remains unchanged and the network function is realized using the same APIs - the neutron service APIs. This helps in keeping the application logic independent of the actual networking hardware.
Additionally, if a vendor wants to provide some additional functionality, he can provide several high-level APIs(2) via the API extension layer. These APIs also interact with the neutron plugin(3) to realize a high-level APIs(2) on the hardware. Using this, a vendor X can also integrate his switches with OpenStack solution. 



Companies are thus integrating their SDN solutions with OpenStack by means of neutron plugin. Refer this link to know more about various solutions available.

Writing OpenStack application - A simple example

Finally, after reading Brent Salisbury's post I got my OpenStack-ODL environment up and running. So, I thought of writing a small application on OpenStack using the novaclient.v1_1.client package.

Setup Environment:

  • As described in this post till the line "The state of OVS after the stack should be the following:" [Just grep this line and do everything done before this.]
  • 2 VM running instances namely test_vm1, test_vm
  • The script should run on the Controller VM.


If you have your own setup, then you can take down the below details

  • Python version 2.7.5
  • Below packages need to be installed
  • pip install python-keystoneclient
  • pip install python-novaclient
Lets look at the below sample script.

  1 #!/usr/bin/env python
  2 import novaclient.v1_1.client as nvclient
  3
  4 # Replace the values below  the ones from your local config,
  5 auth_url = "http://192.168.56.104:5000/v2.0"
  6 username = "admin"
  7 password = "admin"
  8 tenant_name = "demo"
  9 project_id = "demo"
 10
 11
 12 nova = nvclient.Client(auth_url=auth_url, username=username,
 13                            api_key=password, project_id=project_id)
 14
 15 # Get the list of VMs
 16 srv_list =  nova.servers.list()
 17 server = nova.servers.find(name="test_vm1")
 18 print server
 19 print 'Rebooting server '+'test_vm1 ...'
 20 server.reboot()
 21 print 'test_vm1 rebooted'


Line 2 is used for importing nova package with.
Line 5-9 describe your local settings. When you run the controller node using devstack, you'll get the below output:
Horizon is now available at http://192.168.56.104/
Keystone is serving at http://192.168.56.104:5000/v2.0/
The IP Address here is the IP on which horizon service is running.
On line 5, the port number 5000 is used. You may also use port number 35357. Port 5000, 35357 are keystone's port numbers of public and administrative end-points.
For more details on port numbers, check this.

Line 12 is used to create a nova client object against which nova API calls would be made.
Line 16 and 17 are used to obtain the list of VMs and get the object of instance with name "test_vm1".
API call on line 20 finally reboots the instance.
Refer this to see the list of Server module APIs.

*********************************OUTPUT************************************
[fedora@fedora-odl-1 ~]$ python os_sample.py
[, ]
Rebooting server test_vm1 ...
test_vm1 rebooted
[fedora@fedora-odl-1 ~]$


Dashboard state before running the script.

After running the script os_sample, you'll see the rebooting message in the Power state column.

Acknowledgements:
This post is incomplete without acknowledging this wonderful link.

Wednesday, June 22, 2011

Static v/s Dynamic Library

What is a static library?

First of all, a library is a set of functions complied into an object code that is used by applications. When we compile a program that uses a static library, the entire object code gets included in the executable code thus increasing the size of executable. Lets say, the library is altered. Now in this case, you need to recompile the library and recompile your application. The other disadvantage is that if there are several applications running that use the static library, then it leads to wastage of space as the library is included in all the executables. Static libraries are also called as archives(.a extension). Try exploring the makefile, it will have loads of .a libraries in the dependency list.

Dynamic Library:
It consists of routines or functions that are loaded at runtime if a reference exists. In this case, instead of including the entire object code, we include a reference to the library and define the rules on how to find this reference(basically tell the linker where to locate the library for eg, /usr/lib). You can keep it anywhere but just mention the path in the linker search path. They have the advantage that if your library gets changes, the application using the library need not be compiled, it just keeps running. If more than one application using this dynamic library is running, then only a single copy is loaded into the memory which is used by all the applications. These have ".so" extension.