Merge "command not found errors on unstack" into stable/havana

command not found errors on unstack
- add lib/ceilometer and lib/heat to source list for when stop_heat and stop_ceilometer functions are called. - add lib/tls source to lib/keystone for when is_ssl_enabled_service function called. Change-Id: Ief05766e9cfda71fb6392c8a757d04751283414e Closes-Bug: #1269047 (cherry picked from commit 5eec5b6b80)
2014-08-26 00:00:00 +00:00 · 2014-08-25 16:38:26 +08:00 · 2014-08-23 10:01:56 +00:00 · 2014-06-24 11:03:19 -04:00 · 2014-06-23 08:20:02 -04:00 · 2014-06-20 00:30:02 +00:00
57 changed files with 1005 additions and 593 deletions
@@ -2,3 +2,4 @@
 host=review.openstack.org
 port=29418
 project=openstack-dev/devstack.git
+defaultbranch=stable/havana
@@ -48,3 +48,4 @@ Vishvananda Ishaya <vishvananda@gmail.com>
 Yun Mao <yunmao@gmail.com>
 Yong Sheng Gong <gongysh@cn.ibm.com>
 Zhongyue Luo <lzyeval@gmail.com>
+Zhenguo Niu <niu.zglinux@gmail.com>
@@ -5,10 +5,10 @@ Contributing to DevStack
 General
 -------

-DevStack is written in POSIX shell script.  This choice was made because
-it best illustrates the configuration steps that this implementation takes
-on setting up and interacting with OpenStack components.  DevStack specifically
-uses Bash and is compatible with Bash 3.
+DevStack is written in UNIX shell script.  It uses a number of bash-isms
+and so is limited to Bash (version 3 and up) and compatible shells.
+Shell script was chosen because it best illustrates the steps used to
+set up and interact with OpenStack components.

 DevStack's official repository is located on GitHub at
 https://github.com/openstack-dev/devstack.git.  Besides the master branch that
@@ -54,14 +54,14 @@ Sometimes the script needs to know the location of the DevStack install director
 ``TOP_DIR`` should always point there, even if the script itself is located in
 a subdirectory::

-    # Keep track of the current devstack directory.
+    # Keep track of the current DevStack directory.
    TOP_DIR=$(cd $(dirname "$0") && pwd)

 Many scripts will utilize shared functions from the ``functions`` file.  There are
 also rc files (``stackrc`` and ``openrc``) that are often included to set the primary
 configuration of the user environment::

-    # Keep track of the current devstack directory.
+    # Keep track of the current DevStack directory.
    TOP_DIR=$(cd $(dirname "$0") && pwd)

    # Import common functions
@@ -100,13 +100,14 @@ stackrc
 -------

 ``stackrc`` is the global configuration file for DevStack.  It is responsible for
-calling ``localrc`` if it exists so configuration can be overridden by the user.
+calling ``local.conf`` (or ``localrc`` if it exists) so local user configuration
+is recognized.

 The criteria for what belongs in ``stackrc`` can be vaguely summarized as
 follows:

-* All project respositories and branches (for historical reasons)
-* Global configuration that may be referenced in ``localrc``, i.e. ``DEST``, ``DATA_DIR``
+* All project repositories and branches handled directly in ``stack.sh``
+* Global configuration that may be referenced in ``local.conf``, i.e. ``DEST``, ``DATA_DIR``
 * Global service configuration like ``ENABLED_SERVICES``
 * Variables used by multiple services that do not have a clear owner, i.e.
  ``VOLUME_BACKING_FILE_SIZE`` (nova-volumes and cinder) or ``PUBLIC_NETWORK_NAME``
@@ -116,8 +117,9 @@ follows:
  not be changed for other reasons but the earlier file needs to dereference a
  variable set in the later file.  This should be rare.

-Also, variable declarations in ``stackrc`` do NOT allow overriding (the form
-``FOO=${FOO:-baz}``); if they did then they can already be changed in ``localrc``
+Also, variable declarations in ``stackrc`` before ``local.conf`` is sourced
+do NOT allow overriding (the form
+``FOO=${FOO:-baz}``); if they did then they can already be changed in ``local.conf``
 and can stay in the project file.


@@ -139,7 +141,9 @@ verbose in the comments _ABOVE_ the code they pertain to.  Shocco also supports
 Markdown formatting in the comments; use it sparingly.  Specifically, ``stack.sh``
 uses Markdown headers to divide the script into logical sections.

-.. _shocco: http://rtomayko.github.com/shocco/
+.. _shocco: https://github.com/dtroyer/shocco/tree/rst_support
+
+The script used to drive <code>shocco</code> is <code>tools/build_docs.sh</code>.


 Exercises
@@ -6,35 +6,39 @@ DevStack is a set of scripts and utilities to quickly deploy an OpenStack cloud.
 * To describe working configurations of OpenStack (which code branches work together?  what do config files look like for those branches?)
 * To make it easier for developers to dive into OpenStack so that they can productively contribute without having to understand every part of the system at once
 * To make it easy to prototype cross-project features
-* To sanity-check OpenStack builds (used in gating commits to the primary repos)
+* To provide an environment for the OpenStack CI testing on every commit to the projects

-Read more at http://devstack.org (built from the gh-pages branch)
+Read more at http://devstack.org.

-IMPORTANT: Be sure to carefully read `stack.sh` and any other scripts you execute before you run them, as they install software and may alter your networking configuration.  We strongly recommend that you run `stack.sh` in a clean and disposable vm when you are first getting started.
-
-# DevStack on Xenserver
-
-If you would like to use Xenserver as the hypervisor, please refer to the instructions in `./tools/xen/README.md`.
-
-# DevStack on Docker
-
-If you would like to use Docker as the hypervisor, please refer to the instructions in `./tools/docker/README.md`.
+IMPORTANT: Be sure to carefully read `stack.sh` and any other scripts you
+execute before you run them, as they install software and will alter your
+networking configuration.  We strongly recommend that you run `stack.sh`
+in a clean and disposable vm when you are first getting started.

 # Versions

-The devstack master branch generally points to trunk versions of OpenStack components.  For older, stable versions, look for branches named stable/[release] in the DevStack repo.  For example, you can do the following to create a diablo OpenStack cloud:
+The DevStack master branch generally points to trunk versions of OpenStack
+components.  For older, stable versions, look for branches named
+stable/[release] in the DevStack repo.  For example, you can do the
+following to create a grizzly OpenStack cloud:

-    git checkout stable/diablo
+    git checkout stable/grizzly
    ./stack.sh

-You can also pick specific OpenStack project releases by setting the appropriate `*_BRANCH` variables in `localrc` (look in `stackrc` for the default set).  Usually just before a release there will be milestone-proposed branches that need to be tested::
+You can also pick specific OpenStack project releases by setting the appropriate
+`*_BRANCH` variables in the ``localrc`` section of `local.conf` (look in
+`stackrc` for the default set).  Usually just before a release there will be
+milestone-proposed branches that need to be tested::

    GLANCE_REPO=https://github.com/openstack/glance.git
    GLANCE_BRANCH=milestone-proposed

 # Start A Dev Cloud

-Installing in a dedicated disposable vm is safer than installing on your dev machine!  Plus you can pick one of the supported Linux distros for your VM.  To start a dev cloud run the following NOT AS ROOT (see below for more):
+Installing in a dedicated disposable VM is safer than installing on your
+dev machine!  Plus you can pick one of the supported Linux distros for
+your VM.  To start a dev cloud run the following NOT AS ROOT (see
+**DevStack Execution Environment** below for more on user accounts):

    ./stack.sh

@@ -45,7 +49,7 @@ When the script finishes executing, you should be able to access OpenStack endpo

 We also provide an environment file that you can use to interact with your cloud via CLI:

-    # source openrc file to load your environment with osapi and ec2 creds
+    # source openrc file to load your environment with OpenStack CLI creds
    . openrc
    # list instances
    nova list
@@ -61,16 +65,37 @@ If the EC2 API is your cup-o-tea, you can create credentials and use euca2ools:

 DevStack runs rampant over the system it runs on, installing things and uninstalling other things.  Running this on a system you care about is a recipe for disappointment, or worse.  Alas, we're all in the virtualization business here, so run it in a VM.  And take advantage of the snapshot capabilities of your hypervisor of choice to reduce testing cycle times.  You might even save enough time to write one more feature before the next feature freeze...

-``stack.sh`` needs to have root access for a lot of tasks, but it also needs to have not-root permissions for most of its work and for all of the OpenStack services.  So ``stack.sh`` specifically does not run if you are root. This is a recent change (Oct 2013) from the previous behaviour of automatically creating a ``stack`` user.  Automatically creating a user account is not always the right response to running as root, so that bit is now an explicit step using ``tools/create-stack-user.sh``.  Run that (as root!) if you do not want to just use your normal login here, which works perfectly fine.
+``stack.sh`` needs to have root access for a lot of tasks, but uses ``sudo``
+for all of those tasks.  However, it needs to be not-root for most of its
+work and for all of the OpenStack services.  ``stack.sh`` specifically
+does not run if started as root.
+
+This is a recent change (Oct 2013) from the previous behaviour of
+automatically creating a ``stack`` user.  Automatically creating
+user accounts is not the right response to running as root, so
+that bit is now an explicit step using ``tools/create-stack-user.sh``. 
+Run that (as root!) or just check it out to see what DevStack's
+expectations are for the account it runs under.  Many people simply
+use their usual login (the default 'ubuntu' login on a UEC image
+for example).

 # Customizing

-You can override environment variables used in `stack.sh` by creating file name `localrc`.  It is likely that you will need to do this to tweak your networking configuration should you need to access your cloud from a different host.
+You can override environment variables used in `stack.sh` by creating file
+name `local.conf` with a ``locarc`` section as shown below.  It is likely
+that you will need to do this to tweak your networking configuration should
+you need to access your cloud from a different host.
+
+    [[local|localrc]]
+    VARIABLE=value
+
+See the **Local Configuration** section below for more details.

 # Database Backend

 Multiple database backends are available. The available databases are defined in the lib/databases directory.
-`mysql` is the default database, choose a different one by putting the following in `localrc`:
+`mysql` is the default database, choose a different one by putting the
+following in the `localrc` section:

    disable_service mysql
    enable_service postgresql
@@ -81,7 +106,7 @@ Multiple database backends are available. The available databases are defined in

 Multiple RPC backends are available. Currently, this
 includes RabbitMQ (default), Qpid, and ZeroMQ. Your backend of
-choice may be selected via the `localrc`.
+choice may be selected via the `localrc` section.

 Note that selecting more than one RPC backend will result in a failure.

@@ -95,9 +120,10 @@ Example (Qpid):

 # Apache Frontend

-Apache web server is enabled for wsgi services by setting `APACHE_ENABLED_SERVICES` in your localrc. But remember to enable these services at first as above.
+Apache web server is enabled for wsgi services by setting
+`APACHE_ENABLED_SERVICES` in your ``localrc`` section.  Remember to
+enable these services at first as above.

-Example:
    APACHE_ENABLED_SERVICES+=keystone,swift

 # Swift
@@ -108,23 +134,23 @@ vm. When running with only one replica the account, container and
 object services will run directly in screen. The others services like
 replicator, updaters or auditor runs in background.

-If you would like to enable Swift you can add this to your `localrc` :
+If you would like to enable Swift you can add this to your `localrc` section:

    enable_service s-proxy s-object s-container s-account

 If you want a minimal Swift install with only Swift and Keystone you
-can have this instead in your `localrc`:
+can have this instead in your `localrc` section:

    disable_all_services
    enable_service key mysql s-proxy s-object s-container s-account

 If you only want to do some testing of a real normal swift cluster
 with multiple replicas you can do so by customizing the variable
-`SWIFT_REPLICAS` in your `localrc` (usually to 3).
+`SWIFT_REPLICAS` in your `localrc` section (usually to 3).

 # Swift S3

-If you are enabling `swift3` in `ENABLED_SERVICES` devstack will
+If you are enabling `swift3` in `ENABLED_SERVICES` DevStack will
 install the swift3 middleware emulation. Swift will be configured to
 act as a S3 endpoint for Keystone so effectively replacing the
 `nova-objectstore`.
@@ -137,7 +163,7 @@ services are started in background and managed by `swift-init` tool.
 Basic Setup

 In order to enable Neutron a single node setup, you'll need the
-following settings in your `localrc` :
+following settings in your `localrc` section:

    disable_service n-net
    enable_service q-svc
@@ -146,12 +172,15 @@ following settings in your `localrc` :
    enable_service q-l3
    enable_service q-meta
    enable_service neutron
-    # Optional, to enable tempest configuration as part of devstack
+    # Optional, to enable tempest configuration as part of DevStack
    enable_service tempest

 Then run `stack.sh` as normal.

-devstack supports adding specific Neutron configuration flags to the service, Open vSwitch plugin and LinuxBridge plugin configuration files. To make use of this feature, the following variables are defined and can be configured in your `localrc` file:
+DevStack supports setting specific Neutron configuration flags to the
+service, Open vSwitch plugin and LinuxBridge plugin configuration files.
+To make use of this feature, the following variables are defined and can
+be configured in your `localrc` section:

    Variable Name             Config File  Section Modified
    -------------------------------------------------------------------------------------
@@ -160,12 +189,14 @@ devstack supports adding specific Neutron configuration flags to the service, Op
    Q_AGENT_EXTRA_SRV_OPTS    Plugin       `OVS` (for Open Vswitch) or `LINUX_BRIDGE` (for LinuxBridge)
    Q_SRV_EXTRA_DEFAULT_OPTS  Service      DEFAULT

-An example of using the variables in your `localrc` is below:
+An example of using the variables in your `localrc` section is below:

    Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_type=vxlan vxlan_udp_port=8472)
    Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)

-devstack also supports configuring the Neutron ML2 plugin. The ML2 plugin can run with the OVS, LinuxBridge, or Hyper-V agents on compute hosts. A simple way to configure the ml2 plugin is shown below:
+DevStack also supports configuring the Neutron ML2 plugin. The ML2 plugin
+can run with the OVS, LinuxBridge, or Hyper-V agents on compute hosts. A
+simple way to configure the ml2 plugin is shown below:

    # VLAN configuration
    Q_PLUGIN=ml2
@@ -179,7 +210,9 @@ devstack also supports configuring the Neutron ML2 plugin. The ML2 plugin can ru
    Q_PLUGIN=ml2
    Q_ML2_TENANT_NETWORK_TYPE=vxlan

-The above will default in devstack to using the OVS on each compute host. To change this, set the `Q_AGENT` variable to the agent you want to run (e.g. linuxbridge).
+The above will default in DevStack to using the OVS on each compute host.
+To change this, set the `Q_AGENT` variable to the agent you want to run
+(e.g. linuxbridge).

    Variable Name                    Notes
    -------------------------------------------------------------------------------------
@@ -194,13 +227,13 @@ The above will default in devstack to using the OVS on each compute host. To cha
 # Heat

 Heat is disabled by default. To enable it you'll need the following settings
-in your `localrc` :
+in your `localrc` section:

    enable_service heat h-api h-api-cfn h-api-cw h-eng

 Heat can also run in standalone mode, and be configured to orchestrate
 on an external OpenStack cloud. To launch only Heat in standalone mode
-you'll need the following settings in your `localrc` :
+you'll need the following settings in your `localrc` section:

    disable_all_services
    enable_service rabbit mysql heat h-api h-api-cfn h-api-cw h-eng
@@ -215,9 +248,23 @@ If tempest has been successfully configured, a basic set of smoke tests can be r
    $ cd /opt/stack/tempest
    $ nosetests tempest/scenario/test_network_basic_ops.py

+# DevStack on Xenserver
+
+If you would like to use Xenserver as the hypervisor, please refer to the instructions in `./tools/xen/README.md`.
+
+# DevStack on Docker
+
+If you would like to use Docker as the hypervisor, please refer to the instructions in `./tools/docker/README.md`.
+
 # Additional Projects

-DevStack has a hook mechanism to call out to a dispatch script at specific points in the execution if `stack.sh`, `unstack.sh` and `clean.sh`.  This allows higher-level projects, especially those that the lower level projects have no dependency on, to be added to DevStack without modifying the scripts.  Tempest is built this way as an example of how to structure the dispatch script, see `extras.d/80-tempest.sh`.  See `extras.d/README.md` for more information.
+DevStack has a hook mechanism to call out to a dispatch script at specific
+points in the execution of `stack.sh`, `unstack.sh` and `clean.sh`.  This
+allows upper-layer projects, especially those that the lower layer projects
+have no dependency on, to be added to DevStack without modifying the core
+scripts.  Tempest is built this way as an example of how to structure the
+dispatch script, see `extras.d/80-tempest.sh`.  See `extras.d/README.md`
+for more information.

 # Multi-Node Setup

@@ -232,7 +279,8 @@ You should run at least one "controller node", which should have a `stackrc` tha
    enable_service q-meta
    enable_service neutron

-You likely want to change your `localrc` to run a scheduler that will balance VMs across hosts:
+You likely want to change your `localrc` section to run a scheduler that
+will balance VMs across hosts:

    SCHEDULER=nova.scheduler.simple.SimpleScheduler

@@ -249,7 +297,7 @@ You can then run many compute nodes, each of which should have a `stackrc` which

 Cells is a new scaling option with a full spec at http://wiki.openstack.org/blueprint-nova-compute-cells.

-To setup a cells environment add the following to your `localrc`:
+To setup a cells environment add the following to your `localrc` section:

    enable_service n-cell

@@ -264,32 +312,41 @@ Historically DevStack has used ``localrc`` to contain all local configuration an

 The new config file ``local.conf`` is an extended-INI format that introduces a new meta-section header that provides some additional information such as a phase name and destination config filename:

-  [[ <phase> | <filename> ]]
+    [[ <phase> | <config-file-name> ]]

-where <phase> is one of a set of phase names defined by ``stack.sh`` and <filename> is the project config filename.  The filename is eval'ed in the stack.sh context so all environment variables are available and may be used.  Using the project config file variables in the header is strongly suggested (see example of NOVA_CONF below).  If the path of the config file does not exist it is skipped.
+where ``<phase>`` is one of a set of phase names defined by ``stack.sh``
+and ``<config-file-name>`` is the configuration filename.  The filename is
+eval'ed in the ``stack.sh`` context so all environment variables are
+available and may be used.  Using the project config file variables in
+the header is strongly suggested (see the ``NOVA_CONF`` example below).
+If the path of the config file does not exist it is skipped.

 The defined phases are:

-* local - extracts ``localrc`` from ``local.conf`` before ``stackrc`` is sourced
-* post-config - runs after the layer 2 services are configured and before they are started
-* extra - runs after services are started and before any files in ``extra.d`` are executes
+* **local** - extracts ``localrc`` from ``local.conf`` before ``stackrc`` is sourced
+* **post-config** - runs after the layer 2 services are configured and before they are started
+* **extra** - runs after services are started and before any files in ``extra.d`` are executed

 The file is processed strictly in sequence; meta-sections may be specified more than once but if any settings are duplicated the last to appear in the file will be used.

-  [[post-config|$NOVA_CONF]]
-  [DEFAULT]
-  use_syslog = True
+    [[post-config|$NOVA_CONF]]
+    [DEFAULT]
+    use_syslog = True

-  [osapi_v3]
-  enabled = False
+    [osapi_v3]
+    enabled = False

-A specific meta-section ``local:localrc`` is used to provide a default localrc file.  This allows all custom settings for DevStack to be contained in a single file.  ``localrc`` is not overwritten if it exists to preserve compatability.
+A specific meta-section ``local|localrc`` is used to provide a default
+``localrc`` file (actually ``.localrc.auto``).  This allows all custom
+settings for DevStack to be contained in a single file.  If ``localrc``
+exists it will be used instead to preserve backward-compatibility.

-  [[local|localrc]]
-  FIXED_RANGE=10.254.1.0/24
-  ADMIN_PASSWORD=speciale
-  LOGFILE=$DEST/logs/stack.sh.log
+    [[local|localrc]]
+    FIXED_RANGE=10.254.1.0/24
+    ADMIN_PASSWORD=speciale
+    LOGFILE=$DEST/logs/stack.sh.log

-Note that ``Q_PLUGIN_CONF_FILE`` is unique in that it is assumed to _NOT_ start with a ``/`` (slash) character.  A slash will need to be added:
+Note that ``Q_PLUGIN_CONF_FILE`` is unique in that it is assumed to *NOT*
+start with a ``/`` (slash) character.  A slash will need to be added:

-  [[post-config|/$Q_PLUGIN_CONF_FILE]]
+    [[post-config|/$Q_PLUGIN_CONF_FILE]]
@@ -35,6 +35,7 @@ source $TOP_DIR/lib/rpc_backend

 source $TOP_DIR/lib/oslo
 source $TOP_DIR/lib/tls
+
 source $TOP_DIR/lib/horizon
 source $TOP_DIR/lib/keystone
 source $TOP_DIR/lib/glance
@@ -76,7 +77,6 @@ if [[ -d $TOP_DIR/extras.d ]]; then
 fi

 # Clean projects
-cleanup_oslo
 cleanup_cinder
 cleanup_glance
 cleanup_keystone
@@ -100,7 +100,7 @@ META_DATA_2_KEY=foo
 META_DATA_3_KEY=bar

 #ensure no additional metadata is set
-nova aggregate-details $AGGREGATE_ID | egrep "{u'availability_zone': u'$AGGREGATE_A_ZONE'}|{}"
+nova aggregate-details $AGGREGATE_ID | egrep "\|[{u ]*'availability_zone.+$AGGREGATE_A_ZONE'[ }]*\|"

 nova aggregate-set-metadata $AGGREGATE_ID ${META_DATA_1_KEY}=123
 nova aggregate-details $AGGREGATE_ID | grep $META_DATA_1_KEY
@@ -117,7 +117,7 @@ nova aggregate-details $AGGREGATE_ID | grep $META_DATA_3_KEY
 nova aggregate-details $AGGREGATE_ID | grep $META_DATA_2_KEY && die $LINENO "ERROR metadata was not cleared"

 nova aggregate-set-metadata $AGGREGATE_ID $META_DATA_3_KEY $META_DATA_1_KEY
-nova aggregate-details $AGGREGATE_ID | egrep "{u'availability_zone': u'$AGGREGATE_A_ZONE'}|{}"
+nova aggregate-details $AGGREGATE_ID | egrep "\|[{u ]*'availability_zone.+$AGGREGATE_A_ZONE'[ }]*\|"


 # Test aggregate-add/remove-host
@@ -119,7 +119,7 @@ nova flavor-list
 INSTANCE_TYPE=$(nova flavor-list | grep $DEFAULT_INSTANCE_TYPE | get_field 1)
 if [[ -z "$INSTANCE_TYPE" ]]; then
    # grab the first flavor in the list to launch if default doesn't exist
-   INSTANCE_TYPE=$(nova flavor-list | head -n 4 | tail -n 1 | get_field 1)
+    INSTANCE_TYPE=$(nova flavor-list | head -n 4 | tail -n 1 | get_field 1)
 fi

 # Clean-up from previous runs
@@ -62,7 +62,7 @@ die_if_not_set $LINENO IMAGE "Failure getting image $DOCKER_IMAGE_NAME"
 INSTANCE_TYPE=$(nova flavor-list | grep $DEFAULT_INSTANCE_TYPE | get_field 1)
 if [[ -z "$INSTANCE_TYPE" ]]; then
    # grab the first flavor in the list to launch if default doesn't exist
-   INSTANCE_TYPE=$(nova flavor-list | head -n 4 | tail -n 1 | get_field 1)
+    INSTANCE_TYPE=$(nova flavor-list | head -n 4 | tail -n 1 | get_field 1)
 fi

 # Clean-up from previous runs
@@ -102,4 +102,3 @@ set +o xtrace
 echo "*********************************************************************"
 echo "SUCCESS: End DevStack Exercise: $0"
 echo "*********************************************************************"
-
@@ -87,31 +87,31 @@ fi
 # Volumes
 # -------
 if is_service_enabled c-vol && ! is_service_enabled n-cell; then
-   VOLUME_ZONE=`euca-describe-availability-zones | head -n1 | cut -f2`
-   die_if_not_set $LINENO VOLUME_ZONE "Failure to find zone for volume"
+    VOLUME_ZONE=`euca-describe-availability-zones | head -n1 | cut -f2`
+    die_if_not_set $LINENO VOLUME_ZONE "Failure to find zone for volume"

-   VOLUME=`euca-create-volume -s 1 -z $VOLUME_ZONE | cut -f2`
-   die_if_not_set $LINENO VOLUME "Failure to create volume"
+    VOLUME=`euca-create-volume -s 1 -z $VOLUME_ZONE | cut -f2`
+    die_if_not_set $LINENO VOLUME "Failure to create volume"

-   # Test that volume has been created
-   VOLUME=`euca-describe-volumes $VOLUME | cut -f2`
-   die_if_not_set $LINENO VOLUME "Failure to get volume"
+    # Test that volume has been created
+    VOLUME=`euca-describe-volumes $VOLUME | cut -f2`
+    die_if_not_set $LINENO VOLUME "Failure to get volume"

-   # Test volume has become available
-   if ! timeout $RUNNING_TIMEOUT sh -c "while ! euca-describe-volumes $VOLUME | grep -q available; do sleep 1; done"; then
-       die $LINENO "volume didn't become available within $RUNNING_TIMEOUT seconds"
-   fi
+    # Test volume has become available
+    if ! timeout $RUNNING_TIMEOUT sh -c "while ! euca-describe-volumes $VOLUME | grep -q available; do sleep 1; done"; then
+        die $LINENO "volume didn't become available within $RUNNING_TIMEOUT seconds"
+    fi

-   # Attach volume to an instance
-   euca-attach-volume -i $INSTANCE -d $ATTACH_DEVICE $VOLUME || \
-       die $LINENO "Failure attaching volume $VOLUME to $INSTANCE"
-   if ! timeout $ACTIVE_TIMEOUT sh -c "while ! euca-describe-volumes $VOLUME | grep -A 1 in-use | grep -q attach; do sleep 1; done"; then
-       die $LINENO "Could not attach $VOLUME to $INSTANCE"
-   fi
+    # Attach volume to an instance
+    euca-attach-volume -i $INSTANCE -d $ATTACH_DEVICE $VOLUME || \
+        die $LINENO "Failure attaching volume $VOLUME to $INSTANCE"
+    if ! timeout $ACTIVE_TIMEOUT sh -c "while ! euca-describe-volumes $VOLUME | grep -A 1 in-use | grep -q attach; do sleep 1; done"; then
+        die $LINENO "Could not attach $VOLUME to $INSTANCE"
+    fi

-   # Detach volume from an instance
-   euca-detach-volume $VOLUME || \
-       die $LINENO "Failure detaching volume $VOLUME to $INSTANCE"
+    # Detach volume from an instance
+    euca-detach-volume $VOLUME || \
+        die $LINENO "Failure detaching volume $VOLUME to $INSTANCE"
    if ! timeout $ACTIVE_TIMEOUT sh -c "while ! euca-describe-volumes $VOLUME | grep -q available; do sleep 1; done"; then
        die $LINENO "Could not detach $VOLUME to $INSTANCE"
    fi
@@ -120,7 +120,7 @@ if is_service_enabled c-vol && ! is_service_enabled n-cell; then
    euca-delete-volume $VOLUME || \
        die $LINENO "Failure to delete volume"
    if ! timeout $ACTIVE_TIMEOUT sh -c "while euca-describe-volumes | grep $VOLUME; do sleep 1; done"; then
-       die $LINENO "Could not delete $VOLUME"
+        die $LINENO "Could not delete $VOLUME"
    fi
 else
    echo "Volume Tests Skipped"
@@ -113,7 +113,7 @@ nova flavor-list
 INSTANCE_TYPE=$(nova flavor-list | grep $DEFAULT_INSTANCE_TYPE | get_field 1)
 if [[ -z "$INSTANCE_TYPE" ]]; then
    # grab the first flavor in the list to launch if default doesn't exist
-   INSTANCE_TYPE=$(nova flavor-list | head -n 4 | tail -n 1 | get_field 1)
+    INSTANCE_TYPE=$(nova flavor-list | head -n 4 | tail -n 1 | get_field 1)
 fi

 # Clean-up from previous runs
@@ -168,7 +168,7 @@ if ! is_service_enabled neutron; then
    # list floating addresses
    if ! timeout $ASSOCIATE_TIMEOUT sh -c "while ! nova floating-ip-list | grep $TEST_FLOATING_POOL | grep -q $TEST_FLOATING_IP; do sleep 1; done"; then
        die $LINENO "Floating IP not allocated"
-     fi
+    fi
 fi

 # Dis-allow icmp traffic (ping)
@@ -179,6 +179,14 @@ function confirm_server_active {
    fi
 }

+function neutron_debug_admin {
+    local os_username=$OS_USERNAME
+    local os_tenant_id=$OS_TENANT_ID
+    source $TOP_DIR/openrc admin admin
+    neutron-debug $@
+    source $TOP_DIR/openrc $os_username $os_tenant_id
+}
+
 function add_tenant {
    local TENANT=$1
    local USER=$2
@@ -234,7 +242,7 @@ function create_network {
    source $TOP_DIR/openrc $TENANT $TENANT
    local NET_ID=$(neutron net-create --tenant_id $TENANT_ID $NET_NAME $EXTRA| grep ' id ' | awk '{print $4}' )
    neutron subnet-create --ip_version 4 --tenant_id $TENANT_ID --gateway $GATEWAY $NET_ID $CIDR
-    neutron-debug probe-create --device-owner compute $NET_ID
+    neutron_debug_admin probe-create --device-owner compute $NET_ID
    source $TOP_DIR/openrc demo demo
 }

@@ -272,12 +280,12 @@ function create_vms {
 }

 function ping_ip {
-     # Test agent connection.  Assumes namespaces are disabled, and
-     # that DHCP is in use, but not L3
-     local VM_NAME=$1
-     local NET_NAME=$2
-     IP=$(get_instance_ip $VM_NAME $NET_NAME)
-     ping_check $NET_NAME $IP $BOOT_TIMEOUT
+    # Test agent connection.  Assumes namespaces are disabled, and
+    # that DHCP is in use, but not L3
+    local VM_NAME=$1
+    local NET_NAME=$2
+    IP=$(get_instance_ip $VM_NAME $NET_NAME)
+    ping_check $NET_NAME $IP $BOOT_TIMEOUT
 }

 function check_vm {
@@ -329,12 +337,12 @@ function delete_network {
 }

 function delete_networks {
-   foreach_tenant_net 'delete_network ${%TENANT%_NAME} %NUM%'
-   #TODO(nati) add secuirty group check after it is implemented
-   # source $TOP_DIR/openrc demo1 demo1
-   # nova secgroup-delete-rule default icmp -1 -1 0.0.0.0/0
-   # source $TOP_DIR/openrc demo2 demo2
-   # nova secgroup-delete-rule default icmp -1 -1 0.0.0.0/0
+    foreach_tenant_net 'delete_network ${%TENANT%_NAME} %NUM%'
+    # TODO(nati) add secuirty group check after it is implemented
+    # source $TOP_DIR/openrc demo1 demo1
+    # nova secgroup-delete-rule default icmp -1 -1 0.0.0.0/0
+    # source $TOP_DIR/openrc demo2 demo2
+    # nova secgroup-delete-rule default icmp -1 -1 0.0.0.0/0
 }

 function create_all {
@@ -117,7 +117,7 @@ nova flavor-list
 INSTANCE_TYPE=$(nova flavor-list | grep $DEFAULT_INSTANCE_TYPE | get_field 1)
 if [[ -z "$INSTANCE_TYPE" ]]; then
    # grab the first flavor in the list to launch if default doesn't exist
-   INSTANCE_TYPE=$(nova flavor-list | head -n 4 | tail -n 1 | get_field 1)
+    INSTANCE_TYPE=$(nova flavor-list | head -n 4 | tail -n 1 | get_field 1)
 fi

 # Clean-up from previous runs
@@ -10,12 +10,11 @@ that end with `.sh`.  To control the order that the scripts are sourced their
 names start with a two digit sequence number.  DevStack reserves the sequence
 numbers 00 through 09 and 90 through 99 for its own use.

-The scripts are sourced at each hook point so they should not declare anything
-at the top level that would cause a problem, specifically, functions.  This does
-allow the entire `stack.sh` variable space to be available.  The scripts are
+The scripts are sourced at the beginning of each script that calls them. The
+entire `stack.sh` variable space is available.  The scripts are
 sourced with one or more arguments, the first of which defines the hook phase:

-arg 1: source | stack | unstack | clean
+    source | stack | unstack | clean

    source: always called first in any of the scripts, used to set the
        initial defaults in a lib/* script or similar
@@ -0,0 +1 @@
+libxslt1-dev
@@ -66,12 +66,12 @@ fi
 # Heat
 if [[ "$ENABLED_SERVICES" =~ "heat" ]]; then
    HEAT_USER=$(get_id keystone user-create --name=heat \
-                                              --pass="$SERVICE_PASSWORD" \
-                                              --tenant_id $SERVICE_TENANT \
-                                              --email=heat@example.com)
+        --pass="$SERVICE_PASSWORD" \
+        --tenant_id $SERVICE_TENANT \
+        --email=heat@example.com)
    keystone user-role-add --tenant-id $SERVICE_TENANT \
-                           --user-id $HEAT_USER \
-                           --role-id $SERVICE_ROLE
+        --user-id $HEAT_USER \
+        --role-id $SERVICE_ROLE
    # heat_stack_user role is for users created by Heat
    keystone role-create --name heat_stack_user
    if [[ "$KEYSTONE_CATALOG_BACKEND" = 'sql' ]]; then
@@ -126,16 +126,16 @@ fi
 # Ceilometer
 if [[ "$ENABLED_SERVICES" =~ "ceilometer" ]]; then
    CEILOMETER_USER=$(get_id keystone user-create --name=ceilometer \
-                                              --pass="$SERVICE_PASSWORD" \
-                                              --tenant_id $SERVICE_TENANT \
-                                              --email=ceilometer@example.com)
+        --pass="$SERVICE_PASSWORD" \
+        --tenant_id $SERVICE_TENANT \
+        --email=ceilometer@example.com)
    keystone user-role-add --tenant-id $SERVICE_TENANT \
-                           --user-id $CEILOMETER_USER \
-                           --role-id $ADMIN_ROLE
+        --user-id $CEILOMETER_USER \
+        --role-id $ADMIN_ROLE
    # Ceilometer needs ResellerAdmin role to access swift account stats.
    keystone user-role-add --tenant-id $SERVICE_TENANT \
-                           --user-id $CEILOMETER_USER \
-                           --role-id $RESELLER_ROLE
+        --user-id $CEILOMETER_USER \
+        --role-id $RESELLER_ROLE
    if [[ "$KEYSTONE_CATALOG_BACKEND" = 'sql' ]]; then
        CEILOMETER_SERVICE=$(get_id keystone service-create \
            --name=ceilometer \
@@ -0,0 +1 @@
+libxslt-dev
@@ -569,11 +569,14 @@ function git_clone {
    GIT_REF=$3
    RECLONE=$(trueorfalse False $RECLONE)

+    local orig_dir=`pwd`
+
    if [[ "$OFFLINE" = "True" ]]; then
        echo "Running in offline mode, clones already exist"
        # print out the results so we know what change was used in the logs
        cd $GIT_DEST
        git show --oneline | head -1
+        cd $orig_dir
        return
    fi

@@ -622,6 +625,7 @@ function git_clone {
    # print out the results so we know what change was used in the logs
    cd $GIT_DEST
    git show --oneline | head -1
+    cd $orig_dir
 }


@@ -1109,13 +1113,45 @@ function screen_it {
            # sleep to allow bash to be ready to be send the command - we are
            # creating a new window in screen and then sends characters, so if
            # bash isn't running by the time we send the command, nothing happens
-            sleep 1.5
+            sleep 3

            NL=`echo -ne '\015'`
-            screen -S $SCREEN_NAME -p $1 -X stuff "$2 || echo \"$1 failed to start\" | tee \"$SERVICE_DIR/$SCREEN_NAME/$1.failure\"$NL"
+            # This fun command does the following:
+            # - the passed server command is backgrounded
+            # - the pid of the background process is saved in the usual place
+            # - the server process is brought back to the foreground
+            # - if the server process exits prematurely the fg command errors
+            #   and a message is written to stdout and the service failure file
+            # The pid saved can be used in screen_stop() as a process group
+            # id to kill off all child processes
+            screen -S $SCREEN_NAME -p $1 -X stuff "$2 & echo \$! >$SERVICE_DIR/$SCREEN_NAME/$1.pid; fg || echo \"$1 failed to start\" | tee \"$SERVICE_DIR/$SCREEN_NAME/$1.failure\"$NL"
        else
            # Spawn directly without screen
-            run_process "$1" "$2" >$SERVICE_DIR/$SCREEN_NAME/$service.pid
+            run_process "$1" "$2" >$SERVICE_DIR/$SCREEN_NAME/$1.pid
+        fi
+    fi
+}
+
+
+# Stop a service in screen
+# If a PID is available use it, kill the whole process group via TERM
+# If screen is being used kill the screen window; this will catch processes
+# that did not leave a PID behind
+# screen_stop service
+function screen_stop() {
+    SCREEN_NAME=${SCREEN_NAME:-stack}
+    SERVICE_DIR=${SERVICE_DIR:-${DEST}/status}
+    USE_SCREEN=$(trueorfalse True $USE_SCREEN)
+
+    if is_service_enabled $1; then
+        # Kill via pid if we have one available
+        if [[ -r $SERVICE_DIR/$SCREEN_NAME/$1.pid ]]; then
+            pkill -TERM -P -$(cat $SERVICE_DIR/$SCREEN_NAME/$1.pid)
+            rm $SERVICE_DIR/$SCREEN_NAME/$1.pid
+        fi
+        if [[ "$USE_SCREEN" = "True" ]]; then
+            # Clean up the screen window
+            screen -S $SCREEN_NAME -p $1 -X kill
        fi
    fi
 }
@@ -1233,12 +1269,26 @@ function safe_chmod() {
    _safe_permission_operation chmod $@
 }

+# this should be used if you want to install globally, all libraries should
+# use this, especially *oslo* ones
+function setup_install {
+    local project_dir=$1
+    setup_package_with_req_sync $project_dir
+}
+
+# this should be used for projects which run services, like all services
+function setup_develop {
+    local project_dir=$1
+    setup_package_with_req_sync $project_dir -e
+}
+
 # ``pip install -e`` the package, which processes the dependencies
 # using pip before running `setup.py develop`
 # Uses globals ``STACK_USER``, ``TRACK_DEPENDS``, ``REQUIREMENTS_DIR``
 # setup_develop directory
-function setup_develop() {
+function setup_package_with_req_sync() {
    local project_dir=$1
+    local flags=$2

    echo "cd $REQUIREMENTS_DIR; $SUDO_CMD python update.py $project_dir"

@@ -1251,9 +1301,11 @@ function setup_develop() {
            $SUDO_CMD python update.py $project_dir)
    fi

-    pip_install -e $project_dir
+    pip_install $flags $project_dir
    # ensure that further actions can do things like setup.py sdist
-    safe_chown -R $STACK_USER $1/*.egg-info
+    if [[ "$flags" == "-e" ]]; then
+        safe_chown -R $STACK_USER $1/*.egg-info
+    fi

    # Undo requirements changes, if we made them
    if [ $update_requirements -eq 0 ]; then
@@ -1371,9 +1423,9 @@ function upload_image() {
        IMAGE="$FILES/${IMAGE_FNAME}"
        IMAGE_NAME="${IMAGE_FNAME%.xen-raw.tgz}"
        glance \
-          --os-auth-token $token \
-          --os-image-url http://$GLANCE_HOSTPORT \
-          image-create \
+            --os-auth-token $token \
+            --os-image-url http://$GLANCE_HOSTPORT \
+            image-create \
            --name "$IMAGE_NAME" --is-public=True \
            --container-format=tgz --disk-format=raw \
            --property vm_mode=xen < "${IMAGE}"
@@ -1396,11 +1448,11 @@ function upload_image() {
            mkdir "$xdir"
            tar -zxf $FILES/$IMAGE_FNAME -C "$xdir"
            KERNEL=$(for f in "$xdir/"*-vmlinuz* "$xdir/"aki-*/image; do
-                     [ -f "$f" ] && echo "$f" && break; done; true)
+                [ -f "$f" ] && echo "$f" && break; done; true)
            RAMDISK=$(for f in "$xdir/"*-initrd* "$xdir/"ari-*/image; do
-                     [ -f "$f" ] && echo "$f" && break; done; true)
+                [ -f "$f" ] && echo "$f" && break; done; true)
            IMAGE=$(for f in "$xdir/"*.img "$xdir/"ami-*/image; do
-                     [ -f "$f" ] && echo "$f" && break; done; true)
+                [ -f "$f" ] && echo "$f" && break; done; true)
            if [[ -z "$IMAGE_NAME" ]]; then
                IMAGE_NAME=$(basename "$IMAGE" ".img")
            fi
@@ -1435,6 +1487,15 @@ function upload_image() {
            DISK_FORMAT=iso
            CONTAINER_FORMAT=bare
            ;;
+        *.vhd|*.vhdx|*.vhd.gz|*.vhdx.gz)
+            local extension="${IMAGE_FNAME#*.}"
+            IMAGE_NAME=$(basename "$IMAGE" ".$extension")
+            DISK_FORMAT=vhd
+            CONTAINER_FORMAT=bare
+            if [ "${IMAGE_FNAME##*.}" == "gz" ]; then
+                UNPACK=zcat
+            fi
+            ;;
        *) echo "Do not know what to do with $IMAGE_FNAME"; false;;
    esac

@@ -1691,23 +1752,23 @@ function check_path_perm_sanity() {
 #
 # _vercmp_r sep ver1 ver2
 function _vercmp_r {
-  typeset sep
-  typeset -a ver1=() ver2=()
-  sep=$1; shift
-  ver1=("${@:1:sep}")
-  ver2=("${@:sep+1}")
+    typeset sep
+    typeset -a ver1=() ver2=()
+    sep=$1; shift
+    ver1=("${@:1:sep}")
+    ver2=("${@:sep+1}")

-  if ((ver1 > ver2)); then
-    echo 1; return 0
-  elif ((ver2 > ver1)); then
-    echo -1; return 0
-  fi
+    if ((ver1 > ver2)); then
+        echo 1; return 0
+    elif ((ver2 > ver1)); then
+        echo -1; return 0
+    fi

-  if ((sep <= 1)); then
-    echo 0; return 0
-  fi
+    if ((sep <= 1)); then
+        echo 0; return 0
+    fi

-  _vercmp_r $((sep-1)) "${ver1[@]:1}" "${ver2[@]:1}"
+    _vercmp_r $((sep-1)) "${ver1[@]:1}" "${ver2[@]:1}"
 }


@@ -1729,13 +1790,13 @@ function _vercmp_r {
 #
 # vercmp_numbers ver1 ver2
 vercmp_numbers() {
-  typeset v1=$1 v2=$2 sep
-  typeset -a ver1 ver2
+    typeset v1=$1 v2=$2 sep
+    typeset -a ver1 ver2

-  IFS=. read -ra ver1 <<< "$v1"
-  IFS=. read -ra ver2 <<< "$v2"
+    IFS=. read -ra ver1 <<< "$v1"
+    IFS=. read -ra ver2 <<< "$v2"

-  _vercmp_r "${#ver1[@]}" "${ver1[@]}" "${ver2[@]}"
+    _vercmp_r "${#ver1[@]}" "${ver1[@]}" "${ver2[@]}"
 }


@@ -256,19 +256,19 @@ function upload_baremetal_deploy() {

    # load them into glance
    BM_DEPLOY_KERNEL_ID=$(glance \
-         --os-auth-token $token \
-         --os-image-url http://$GLANCE_HOSTPORT \
-         image-create \
-         --name $BM_DEPLOY_KERNEL \
-         --is-public True --disk-format=aki \
-         < $TOP_DIR/files/$BM_DEPLOY_KERNEL  | grep ' id ' | get_field 2)
+        --os-auth-token $token \
+        --os-image-url http://$GLANCE_HOSTPORT \
+        image-create \
+        --name $BM_DEPLOY_KERNEL \
+        --is-public True --disk-format=aki \
+        < $TOP_DIR/files/$BM_DEPLOY_KERNEL  | grep ' id ' | get_field 2)
    BM_DEPLOY_RAMDISK_ID=$(glance \
-         --os-auth-token $token \
-         --os-image-url http://$GLANCE_HOSTPORT \
-         image-create \
-         --name $BM_DEPLOY_RAMDISK \
-         --is-public True --disk-format=ari \
-         < $TOP_DIR/files/$BM_DEPLOY_RAMDISK  | grep ' id ' | get_field 2)
+        --os-auth-token $token \
+        --os-image-url http://$GLANCE_HOSTPORT \
+        image-create \
+        --name $BM_DEPLOY_RAMDISK \
+        --is-public True --disk-format=ari \
+        < $TOP_DIR/files/$BM_DEPLOY_RAMDISK  | grep ' id ' | get_field 2)
 }

 # create a basic baremetal flavor, associated with deploy kernel & ramdisk
@@ -278,11 +278,11 @@ function create_baremetal_flavor() {
    aki=$1
    ari=$2
    nova flavor-create $BM_FLAVOR_NAME $BM_FLAVOR_ID \
-            $BM_FLAVOR_RAM $BM_FLAVOR_ROOT_DISK $BM_FLAVOR_CPU
+        $BM_FLAVOR_RAM $BM_FLAVOR_ROOT_DISK $BM_FLAVOR_CPU
    nova flavor-key $BM_FLAVOR_NAME set \
-            "cpu_arch"="$BM_FLAVOR_ARCH" \
-            "baremetal:deploy_kernel_id"="$aki" \
-            "baremetal:deploy_ramdisk_id"="$ari"
+        "cpu_arch"="$BM_FLAVOR_ARCH" \
+        "baremetal:deploy_kernel_id"="$aki" \
+        "baremetal:deploy_ramdisk_id"="$ari"

 }

@@ -311,19 +311,19 @@ function extract_and_upload_k_and_r_from_image() {

    # load them into glance
    KERNEL_ID=$(glance \
-         --os-auth-token $token \
-         --os-image-url http://$GLANCE_HOSTPORT \
-         image-create \
-         --name $image_name-kernel \
-         --is-public True --disk-format=aki \
-         < $TOP_DIR/files/$OUT_KERNEL | grep ' id ' | get_field 2)
+        --os-auth-token $token \
+        --os-image-url http://$GLANCE_HOSTPORT \
+        image-create \
+        --name $image_name-kernel \
+        --is-public True --disk-format=aki \
+        < $TOP_DIR/files/$OUT_KERNEL | grep ' id ' | get_field 2)
    RAMDISK_ID=$(glance \
-         --os-auth-token $token \
-         --os-image-url http://$GLANCE_HOSTPORT \
-         image-create \
-         --name $image_name-initrd \
-         --is-public True --disk-format=ari \
-         < $TOP_DIR/files/$OUT_RAMDISK | grep ' id ' | get_field 2)
+        --os-auth-token $token \
+        --os-image-url http://$GLANCE_HOSTPORT \
+        image-create \
+        --name $image_name-initrd \
+        --is-public True --disk-format=ari \
+        < $TOP_DIR/files/$OUT_RAMDISK | grep ' id ' | get_field 2)
 }


@@ -365,11 +365,11 @@ function upload_baremetal_image() {
            mkdir "$xdir"
            tar -zxf $FILES/$IMAGE_FNAME -C "$xdir"
            KERNEL=$(for f in "$xdir/"*-vmlinuz* "$xdir/"aki-*/image; do
-                     [ -f "$f" ] && echo "$f" && break; done; true)
+                [ -f "$f" ] && echo "$f" && break; done; true)
            RAMDISK=$(for f in "$xdir/"*-initrd* "$xdir/"ari-*/image; do
-                     [ -f "$f" ] && echo "$f" && break; done; true)
+                [ -f "$f" ] && echo "$f" && break; done; true)
            IMAGE=$(for f in "$xdir/"*.img "$xdir/"ami-*/image; do
-                     [ -f "$f" ] && echo "$f" && break; done; true)
+                [ -f "$f" ] && echo "$f" && break; done; true)
            if [[ -z "$IMAGE_NAME" ]]; then
                IMAGE_NAME=$(basename "$IMAGE" ".img")
            fi
@@ -403,19 +403,19 @@ function upload_baremetal_image() {
            --container-format ari \
            --disk-format ari < "$RAMDISK" | grep ' id ' | get_field 2)
    else
-       # TODO(deva): add support for other image types
-       return
+        # TODO(deva): add support for other image types
+        return
    fi

    glance \
-       --os-auth-token $token \
-       --os-image-url http://$GLANCE_HOSTPORT \
-       image-create \
-       --name "${IMAGE_NAME%.img}" --is-public True \
-       --container-format $CONTAINER_FORMAT \
-       --disk-format $DISK_FORMAT \
-       ${KERNEL_ID:+--property kernel_id=$KERNEL_ID} \
-       ${RAMDISK_ID:+--property ramdisk_id=$RAMDISK_ID} < "${IMAGE}"
+        --os-auth-token $token \
+        --os-image-url http://$GLANCE_HOSTPORT \
+        image-create \
+        --name "${IMAGE_NAME%.img}" --is-public True \
+        --container-format $CONTAINER_FORMAT \
+        --disk-format $DISK_FORMAT \
+        ${KERNEL_ID:+--property kernel_id=$KERNEL_ID} \
+        ${RAMDISK_ID:+--property ramdisk_id=$RAMDISK_ID} < "${IMAGE}"

    # override DEFAULT_IMAGE_NAME so that tempest can find the image
    # that we just uploaded in glance
@@ -439,15 +439,15 @@ function add_baremetal_node() {
    mac_2=${2:-$BM_SECOND_MAC}

    id=$(nova baremetal-node-create \
-       --pm_address="$BM_PM_ADDR" \
-       --pm_user="$BM_PM_USER" \
-       --pm_password="$BM_PM_PASS" \
-       "$BM_HOSTNAME" \
-       "$BM_FLAVOR_CPU" \
-       "$BM_FLAVOR_RAM" \
-       "$BM_FLAVOR_ROOT_DISK" \
-       "$mac_1" \
-       | grep ' id ' | get_field 2 )
+        --pm_address="$BM_PM_ADDR" \
+        --pm_user="$BM_PM_USER" \
+        --pm_password="$BM_PM_PASS" \
+        "$BM_HOSTNAME" \
+        "$BM_FLAVOR_CPU" \
+        "$BM_FLAVOR_RAM" \
+        "$BM_FLAVOR_ROOT_DISK" \
+        "$mac_1" \
+        | grep ' id ' | get_field 2 )
    [ $? -eq 0 ] || [ "$id" ] || die $LINENO "Error adding baremetal node"
    if [ -n "$mac_2" ]; then
        id2=$(nova baremetal-interface-add "$id" "$mac_2" )
@@ -146,7 +146,7 @@ function start_ceilometer() {
 function stop_ceilometer() {
    # Kill the ceilometer screen windows
    for serv in ceilometer-acompute ceilometer-acentral ceilometer-collector ceilometer-api ceilometer-alarm-notifier ceilometer-alarm-evaluator; do
-        screen -S $SCREEN_NAME -p $serv -X kill
+        screen_stop $serv
    done
 }

@@ -202,15 +202,25 @@ function configure_cinder() {
    sudo mv $TEMPFILE /etc/sudoers.d/cinder-rootwrap

    cp $CINDER_DIR/etc/cinder/api-paste.ini $CINDER_API_PASTE_INI
-    iniset $CINDER_API_PASTE_INI filter:authtoken auth_host $KEYSTONE_AUTH_HOST
-    iniset $CINDER_API_PASTE_INI filter:authtoken auth_port $KEYSTONE_AUTH_PORT
-    iniset $CINDER_API_PASTE_INI filter:authtoken auth_protocol $KEYSTONE_AUTH_PROTOCOL
-    iniset $CINDER_API_PASTE_INI filter:authtoken admin_tenant_name $SERVICE_TENANT_NAME
-    iniset $CINDER_API_PASTE_INI filter:authtoken admin_user cinder
-    iniset $CINDER_API_PASTE_INI filter:authtoken admin_password $SERVICE_PASSWORD
-    iniset $CINDER_API_PASTE_INI filter:authtoken signing_dir $CINDER_AUTH_CACHE_DIR
+
+    inicomment $CINDER_API_PASTE_INI filter:authtoken auth_host
+    inicomment $CINDER_API_PASTE_INI filter:authtoken auth_port
+    inicomment $CINDER_API_PASTE_INI filter:authtoken auth_protocol
+    inicomment $CINDER_API_PASTE_INI filter:authtoken admin_tenant_name
+    inicomment $CINDER_API_PASTE_INI filter:authtoken admin_user
+    inicomment $CINDER_API_PASTE_INI filter:authtoken admin_password
+    inicomment $CINDER_API_PASTE_INI filter:authtoken signing_dir

    cp $CINDER_DIR/etc/cinder/cinder.conf.sample $CINDER_CONF
+
+    iniset $CINDER_CONF keystone_authtoken auth_host $KEYSTONE_AUTH_HOST
+    iniset $CINDER_CONF keystone_authtoken auth_port $KEYSTONE_AUTH_PORT
+    iniset $CINDER_CONF keystone_authtoken auth_protocol $KEYSTONE_AUTH_PROTOCOL
+    iniset $CINDER_CONF keystone_authtoken admin_tenant_name $SERVICE_TENANT_NAME
+    iniset $CINDER_CONF keystone_authtoken admin_user cinder
+    iniset $CINDER_CONF keystone_authtoken admin_password $SERVICE_PASSWORD
+    iniset $CINDER_CONF keystone_authtoken signing_dir $CINDER_AUTH_CACHE_DIR
+
    iniset $CINDER_CONF DEFAULT auth_strategy keystone
    iniset $CINDER_CONF DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL
    iniset $CINDER_CONF DEFAULT verbose True
@@ -520,7 +530,7 @@ function start_cinder() {
 function stop_cinder() {
    # Kill the cinder screen windows
    for serv in c-api c-bak c-sch c-vol; do
-        screen -S $SCREEN_NAME -p $serv -X kill
+        screen_stop $serv
    done

    if is_service_enabled c-vol; then
@@ -10,7 +10,7 @@
 #   [[group-name|file-name]]
 #
 # group-name refers to the group of configuration file changes to be processed
-# at a particular time.  These are called phases in ``stack.sh`` but 
+# at a particular time.  These are called phases in ``stack.sh`` but
 # group here as these functions are not DevStack-specific.
 #
 # file-name is the destination of the config file
@@ -64,12 +64,12 @@ function get_meta_section_files() {
    [[ -r $file ]] || return 0

    $CONFIG_AWK_CMD -v matchgroup=$matchgroup '
-      /^\[\[.+\|.*\]\]/ {
-          gsub("[][]", "", $1);
-          split($1, a, "|");
-          if (a[1] == matchgroup)
-              print a[2]
-      }
+        /^\[\[.+\|.*\]\]/ {
+            gsub("[][]", "", $1);
+            split($1, a, "|");
+            if (a[1] == matchgroup)
+                print a[2]
+        }
    ' $file
 }

@@ -121,6 +121,8 @@ function configure_glance() {
        iniset $GLANCE_API_CONF DEFAULT swift_store_user $SERVICE_TENANT_NAME:glance
        iniset $GLANCE_API_CONF DEFAULT swift_store_key $SERVICE_PASSWORD
        iniset $GLANCE_API_CONF DEFAULT swift_store_create_container_on_put True
+
+        iniset $GLANCE_API_CONF DEFAULT known_stores "glance.store.filesystem.Store, glance.store.http.Store, glance.store.swift.Store"
    fi

    cp -p $GLANCE_DIR/etc/glance-registry-paste.ini $GLANCE_REGISTRY_PASTE_INI
@@ -194,15 +196,15 @@ function start_glance() {
    screen_it g-api "cd $GLANCE_DIR; $GLANCE_BIN_DIR/glance-api --config-file=$GLANCE_CONF_DIR/glance-api.conf"
    echo "Waiting for g-api ($GLANCE_HOSTPORT) to start..."
    if ! timeout $SERVICE_TIMEOUT sh -c "while ! wget --no-proxy -q -O- http://$GLANCE_HOSTPORT; do sleep 1; done"; then
-      die $LINENO "g-api did not start"
+        die $LINENO "g-api did not start"
    fi
 }

 # stop_glance() - Stop running processes
 function stop_glance() {
    # Kill the Glance screen windows
-    screen -S $SCREEN_NAME -p g-api -X kill
-    screen -S $SCREEN_NAME -p g-reg -X kill
+    screen_stop g-api
+    screen_stop g-reg
 }


@@ -175,7 +175,7 @@ function start_heat() {
 function stop_heat() {
    # Kill the screen windows
    for serv in h-eng h-api h-api-cfn h-api-cw; do
-        screen -S $SCREEN_NAME -p $serv -X kill
+        screen_stop $serv
    done
 }

@@ -29,6 +29,7 @@ set +o xtrace

 # Set up default directories
 HORIZON_DIR=$DEST/horizon
+HORIZONAUTH_DIR=$DEST/django_openstack_auth

 # local_settings.py is used to customize Dashboard settings.
 # The example file in Horizon repo is used by default.
@@ -151,6 +152,12 @@ function init_horizon() {
    \" $FILES/apache-horizon.template >$horizon_conf"
 }

+# install_django_openstack_auth() - Collect source and prepare
+function install_django_openstack_auth {
+    git_clone $HORIZONAUTH_REPO $HORIZONAUTH_DIR $HORIZONAUTH_BRANCH
+    setup_install $HORIZONAUTH_DIR
+}
+
 # install_horizon() - Collect source and prepare
 function install_horizon() {
    # Apache installation, because we mark it NOPRIME
@@ -45,7 +45,7 @@ function install_infra() {

    # Install pbr
    git_clone $PBR_REPO $PBR_DIR $PBR_BRANCH
-    setup_develop $PBR_DIR
+    setup_install $PBR_DIR
 }

 # Restore xtrace
@@ -195,7 +195,7 @@ function start_ironic_api() {
    screen_it ir-api "cd $IRONIC_DIR; $IRONIC_BIN_DIR/ironic-api --config-file=$IRONIC_CONF_FILE"
    echo "Waiting for ir-api ($IRONIC_HOSTPORT) to start..."
    if ! timeout $SERVICE_TIMEOUT sh -c "while ! wget --no-proxy -q -O- http://$IRONIC_HOSTPORT; do sleep 1; done"; then
-      die $LINENO "ir-api did not start"
+        die $LINENO "ir-api did not start"
    fi
 }

@@ -26,6 +26,7 @@
 XTRACE=$(set +o | grep xtrace)
 set +o xtrace

+source $TOP_DIR/lib/tls

 # Defaults
 # --------
@@ -69,6 +70,8 @@ KEYSTONE_SERVICE_PORT=${KEYSTONE_SERVICE_PORT:-5000}
 KEYSTONE_SERVICE_PORT_INT=${KEYSTONE_SERVICE_PORT_INT:-5001}
 KEYSTONE_SERVICE_PROTOCOL=${KEYSTONE_SERVICE_PROTOCOL:-$SERVICE_PROTOCOL}

+# Bind hosts
+KEYSTONE_ADMIN_BIND_HOST=${KEYSTONE_ADMIN_BIND_HOST:-$KEYSTONE_SERVICE_HOST}
 # Set the tenant for service accounts in Keystone
 SERVICE_TENANT_NAME=${SERVICE_TENANT_NAME:-service}

@@ -169,6 +172,7 @@ function configure_keystone() {
    # Set the URL advertised in the ``versions`` structure returned by the '/' route
    iniset $KEYSTONE_CONF DEFAULT public_endpoint "$KEYSTONE_SERVICE_PROTOCOL://$KEYSTONE_SERVICE_HOST:%(public_port)s/"
    iniset $KEYSTONE_CONF DEFAULT admin_endpoint "$KEYSTONE_SERVICE_PROTOCOL://$KEYSTONE_SERVICE_HOST:%(admin_port)s/"
+    iniset $KEYSTONE_CONF DEFAULT admin_bind_host "$KEYSTONE_ADMIN_BIND_HOST"

    if is_service_enabled tls-proxy; then
        # Set the service ports for a proxy to take the originals
@@ -373,7 +377,7 @@ function start_keystone() {

    echo "Waiting for keystone to start..."
    if ! timeout $SERVICE_TIMEOUT sh -c "while ! curl --noproxy '*' -s http://$SERVICE_HOST:$service_port/v$IDENTITY_API_VERSION/ >/dev/null; do sleep 1; done"; then
-      die $LINENO "keystone did not start"
+        die $LINENO "keystone did not start"
    fi

    # Start proxies if enabled
@@ -386,7 +390,7 @@ function start_keystone() {
 # stop_keystone() - Stop running processes
 function stop_keystone() {
    # Kill the Keystone screen window
-    screen -S $SCREEN_NAME -p key -X kill
+    screen_stop key
 }


@@ -79,8 +79,8 @@ NEUTRON_AUTH_CACHE_DIR=${NEUTRON_AUTH_CACHE_DIR:-/var/cache/neutron}
 # Support entry points installation of console scripts
 if [[ -d $NEUTRON_DIR/bin/neutron-server ]]; then
    NEUTRON_BIN_DIR=$NEUTRON_DIR/bin
-     else
-NEUTRON_BIN_DIR=$(get_python_exec_prefix)
+else
+    NEUTRON_BIN_DIR=$(get_python_exec_prefix)
 fi

 NEUTRON_CONF_DIR=/etc/neutron
@@ -110,6 +110,15 @@ Q_ALLOW_OVERLAPPING_IP=${Q_ALLOW_OVERLAPPING_IP:-True}
 Q_USE_DEBUG_COMMAND=${Q_USE_DEBUG_COMMAND:-False}
 # The name of the default q-l3 router
 Q_ROUTER_NAME=${Q_ROUTER_NAME:-router1}
+
+# The next two variables are configured by plugin
+# e.g.  _configure_neutron_l3_agent or lib/neutron_plugins/*
+#
+# The plugin supports L3.
+Q_L3_ENABLED=${Q_L3_ENABLED:-False}
+# L3 routers exist per tenant
+Q_L3_ROUTER_PER_TENANT=${Q_L3_ROUTER_PER_TENANT:-False}
+
 # List of config file names in addition to the main plugin config file
 # See _configure_neutron_common() for details about setting it up
 declare -a Q_PLUGIN_EXTRA_CONF_FILES
@@ -373,7 +382,7 @@ function create_neutron_initial_network() {
                iniset $Q_L3_CONF_FILE DEFAULT router_id $ROUTER_ID
            fi
        fi
-   fi
+    fi
 }

 # init_neutron() - Initialize databases, etc.
@@ -404,7 +413,7 @@ function install_neutron_agent_packages() {
    fi

    if is_service_enabled q-lbaas; then
-       neutron_agent_lbaas_install_agent_packages
+        neutron_agent_lbaas_install_agent_packages
    fi
 }

@@ -414,13 +423,13 @@ function start_neutron_service_and_check() {
    local cfg_file
    local CFG_FILE_OPTIONS="--config-file $NEUTRON_CONF --config-file /$Q_PLUGIN_CONF_FILE"
    for cfg_file in ${Q_PLUGIN_EXTRA_CONF_FILES[@]}; do
-         CFG_FILE_OPTIONS+=" --config-file /$cfg_file"
+        CFG_FILE_OPTIONS+=" --config-file /$cfg_file"
    done
    # Start the Neutron service
    screen_it q-svc "cd $NEUTRON_DIR && python $NEUTRON_BIN_DIR/neutron-server $CFG_FILE_OPTIONS"
    echo "Waiting for Neutron to start..."
    if ! timeout $SERVICE_TIMEOUT sh -c "while ! wget --no-proxy -q -O- http://$Q_HOST:$Q_PORT; do sleep 1; done"; then
-      die $LINENO "Neutron did not start"
+        die $LINENO "Neutron did not start"
    fi
 }

@@ -460,8 +469,7 @@ function stop_neutron() {
        [ ! -z "$pid" ] && sudo kill -9 $pid
    fi
    if is_service_enabled q-meta; then
-        pid=$(ps aux | awk '/neutron-ns-metadata-proxy/ { print $2 }')
-        [ ! -z "$pid" ] && sudo kill -9 $pid
+        sudo pkill -9 -f neutron-ns-metadata-proxy || :
    fi
 }

@@ -712,9 +720,9 @@ function _neutron_setup_rootwrap() {
    # Set up ``rootwrap.conf``, pointing to ``$NEUTRON_CONF_DIR/rootwrap.d``
    # location moved in newer versions, prefer new location
    if test -r $NEUTRON_DIR/etc/neutron/rootwrap.conf; then
-      sudo cp -p $NEUTRON_DIR/etc/neutron/rootwrap.conf $Q_RR_CONF_FILE
+        sudo cp -p $NEUTRON_DIR/etc/neutron/rootwrap.conf $Q_RR_CONF_FILE
    else
-      sudo cp -p $NEUTRON_DIR/etc/rootwrap.conf $Q_RR_CONF_FILE
+        sudo cp -p $NEUTRON_DIR/etc/rootwrap.conf $Q_RR_CONF_FILE
    fi
    sudo sed -e "s:^filters_path=.*$:filters_path=$Q_CONF_ROOTWRAP_D:" -i $Q_RR_CONF_FILE
    sudo chown root:root $Q_RR_CONF_FILE
@@ -848,11 +856,11 @@ function _ssh_check_neutron() {
 # please refer to ``lib/neutron_thirdparty/README.md`` for details
 NEUTRON_THIRD_PARTIES=""
 for f in $TOP_DIR/lib/neutron_thirdparty/*; do
-     third_party=$(basename $f)
-     if is_service_enabled $third_party; then
-         source $TOP_DIR/lib/neutron_thirdparty/$third_party
-         NEUTRON_THIRD_PARTIES="$NEUTRON_THIRD_PARTIES,$third_party"
-     fi
+    third_party=$(basename $f)
+    if is_service_enabled $third_party; then
+        source $TOP_DIR/lib/neutron_thirdparty/$third_party
+        NEUTRON_THIRD_PARTIES="$NEUTRON_THIRD_PARTIES,$third_party"
+    fi
 done

 function _neutron_third_party_do() {
@@ -32,19 +32,18 @@ function neutron_plugin_configure_debug_command() {

 function neutron_plugin_configure_dhcp_agent() {
    DHCP_DRIVER=${DHCP_DRIVER:-"neutron.plugins.midonet.agent.midonet_driver.DhcpNoOpDriver"}
-    DHCP_INTERFACE_DRIVER=${DHCP_INTEFACE_DRIVER:-"neutron.plugins.midonet.agent.midonet_driver.MidonetInterfaceDriver"}
+    neutron_plugin_setup_interface_driver $Q_DHCP_CONF_FILE
    iniset $Q_DHCP_CONF_FILE DEFAULT dhcp_driver $DHCP_DRIVER
-    iniset $Q_DHCP_CONF_FILE DEFAULT interface_driver $DHCP_INTERFACE_DRIVER
    iniset $Q_DHCP_CONF_FILE DEFAULT use_namespaces True
    iniset $Q_DHCP_CONF_FILE DEFAULT enable_isolated_metadata True
 }

 function neutron_plugin_configure_l3_agent() {
-   die $LINENO "q-l3 must not be executed with MidoNet plugin!"
+    die $LINENO "q-l3 must not be executed with MidoNet plugin!"
 }

 function neutron_plugin_configure_plugin_agent() {
-   die $LINENO "q-agt must not be executed with MidoNet plugin!"
+    die $LINENO "q-agt must not be executed with MidoNet plugin!"
 }

 function neutron_plugin_configure_service() {
@@ -66,8 +65,8 @@ function neutron_plugin_configure_service() {
 }

 function neutron_plugin_setup_interface_driver() {
-    # May change in the future
-    :
+    local conf_file=$1
+    iniset $conf_file DEFAULT interface_driver neutron.agent.linux.interface.MidonetInterfaceDriver
 }

 function has_neutron_plugin_security_group() {
@@ -101,15 +101,15 @@ function _neutron_setup_ovs_tunnels() {
    local id=0
    GRE_LOCAL_IP=${GRE_LOCAL_IP:-$HOST_IP}
    if [ -n "$GRE_REMOTE_IPS" ]; then
-         for ip in ${GRE_REMOTE_IPS//:/ }
-         do
-             if [[ "$ip" == "$GRE_LOCAL_IP" ]]; then
-                 continue
-             fi
-             sudo ovs-vsctl --no-wait add-port $bridge gre$id -- \
-                 set Interface gre$id type=gre options:remote_ip=$ip
-             id=`expr $id + 1`
-         done
+        for ip in ${GRE_REMOTE_IPS//:/ }
+        do
+            if [[ "$ip" == "$GRE_LOCAL_IP" ]]; then
+                continue
+            fi
+            sudo ovs-vsctl --no-wait add-port $bridge gre$id -- \
+                set Interface gre$id type=gre options:remote_ip=$ip
+            id=`expr $id + 1`
+        done
    fi
 }

@@ -58,13 +58,13 @@ function neutron_plugin_configure_dhcp_agent() {
 }

 function neutron_plugin_configure_l3_agent() {
-   # Nicira plugin does not run L3 agent
-   die $LINENO "q-l3 should must not be executed with Nicira plugin!"
+    # Nicira plugin does not run L3 agent
+    die $LINENO "q-l3 should must not be executed with Nicira plugin!"
 }

 function neutron_plugin_configure_plugin_agent() {
-   # Nicira plugin does not run L2 agent
-   die $LINENO "q-agt must not be executed with Nicira plugin!"
+    # Nicira plugin does not run L2 agent
+    die $LINENO "q-agt must not be executed with Nicira plugin!"
 }

 function neutron_plugin_configure_service() {
@@ -6,8 +6,6 @@
 MY_XTRACE=$(set +o | grep xtrace)
 set +o xtrace

-#source $TOP_DIR/lib/neutron_plugins/ovs_base
-
 function neutron_plugin_create_nova_conf() {

    NOVA_VIF_DRIVER=${NOVA_VIF_DRIVER:-"nova.virt.libvirt.vif.LibvirtGenericVIFDriver"}
@@ -24,11 +22,17 @@ function neutron_plugin_configure_common() {
    Q_PLUGIN_CLASS="neutron.plugins.plumgrid.plumgrid_plugin.plumgrid_plugin.NeutronPluginPLUMgridV2"
    PLUMGRID_DIRECTOR_IP=${PLUMGRID_DIRECTOR_IP:-localhost}
    PLUMGRID_DIRECTOR_PORT=${PLUMGRID_DIRECTOR_PORT:-7766}
+    PLUMGRID_ADMIN=${PLUMGRID_ADMIN:-username}
+    PLUMGRID_PASSWORD=${PLUMGRID_PASSWORD:-password}
+    PLUMGRID_TIMEOUT=${PLUMGRID_TIMEOUT:-70}
 }

 function neutron_plugin_configure_service() {
    iniset /$Q_PLUGIN_CONF_FILE PLUMgridDirector director_server $PLUMGRID_DIRECTOR_IP
    iniset /$Q_PLUGIN_CONF_FILE PLUMgridDirector director_server_port $PLUMGRID_DIRECTOR_PORT
+    iniset /$Q_PLUGIN_CONF_FILE PLUMgridDirector username $PLUMGRID_ADMIN
+    iniset /$Q_PLUGIN_CONF_FILE PLUMgridDirector password $PLUMGRID_PASSWORD
+    iniset /$Q_PLUGIN_CONF_FILE PLUMgridDirector servertimeout $PLUMGRID_TIMEOUT
 }

 function neutron_plugin_configure_debug_command() {
@@ -66,8 +66,8 @@ function init_trema() {

    cp $TREMA_SS_DIR/sliceable_switch_null.conf $TREMA_SS_CONFIG
    sed -i -e "s|^\$apps_dir.*$|\$apps_dir = \"$TREMA_DIR/apps\"|" \
-           -e "s|^\$db_dir.*$|\$db_dir = \"$TREMA_SS_DB_DIR\"|" \
-           $TREMA_SS_CONFIG
+        -e "s|^\$db_dir.*$|\$db_dir = \"$TREMA_SS_DB_DIR\"|" \
+        $TREMA_SS_CONFIG
 }

 function gem_install() {
@@ -212,26 +212,24 @@ function configure_nova() {
    configure_nova_rootwrap

    if is_service_enabled n-api; then
-        # Use the sample http middleware configuration supplied in the
-        # Nova sources.  This paste config adds the configuration required
-        # for Nova to validate Keystone tokens.
-
        # Remove legacy paste config if present
        rm -f $NOVA_DIR/bin/nova-api-paste.ini

        # Get the sample configuration file in place
        cp $NOVA_DIR/etc/nova/api-paste.ini $NOVA_CONF_DIR

-        iniset $NOVA_API_PASTE_INI filter:authtoken auth_host $KEYSTONE_AUTH_HOST
+        # Comment out the keystone configs in Nova's api-paste.ini.
+        # We are using nova.conf to configure this instead.
+        inicomment $NOVA_API_PASTE_INI filter:authtoken auth_host
        if is_service_enabled tls-proxy; then
-            iniset $NOVA_API_PASTE_INI filter:authtoken auth_protocol $KEYSTONE_AUTH_PROTOCOL
+            inicomment $NOVA_API_PASTE_INI filter:authtoken auth_protocol
        fi
-        iniset $NOVA_API_PASTE_INI filter:authtoken admin_tenant_name $SERVICE_TENANT_NAME
-        iniset $NOVA_API_PASTE_INI filter:authtoken admin_user nova
-        iniset $NOVA_API_PASTE_INI filter:authtoken admin_password $SERVICE_PASSWORD
+        inicomment $NOVA_API_PASTE_INI filter:authtoken admin_tenant_name
+        inicomment $NOVA_API_PASTE_INI filter:authtoken admin_user
+        inicomment $NOVA_API_PASTE_INI filter:authtoken admin_password
    fi

-    iniset $NOVA_API_PASTE_INI filter:authtoken signing_dir $NOVA_AUTH_CACHE_DIR
+    inicomment $NOVA_API_PASTE_INI filter:authtoken signing_dir

    if is_service_enabled n-cpu; then
        # Force IP forwarding on, just on case
@@ -394,7 +392,20 @@ function create_nova_conf() {
            # Set the service port for a proxy to take the original
            iniset $NOVA_CONF DEFAULT osapi_compute_listen_port "$NOVA_SERVICE_PORT_INT"
        fi
+
+        # Add keystone authtoken configuration
+
+        iniset $NOVA_CONF keystone_authtoken auth_host $KEYSTONE_AUTH_HOST
+        if is_service_enabled tls-proxy; then
+            iniset $NOVA_CONF keystone_authtoken auth_protocol $KEYSTONE_AUTH_PROTOCOL
+        fi
+        iniset $NOVA_CONF keystone_authtoken admin_tenant_name $SERVICE_TENANT_NAME
+        iniset $NOVA_CONF keystone_authtoken admin_user nova
+        iniset $NOVA_CONF keystone_authtoken admin_password $SERVICE_PASSWORD
    fi
+
+    iniset $NOVA_CONF keystone_authtoken signing_dir $NOVA_AUTH_CACHE_DIR
+
    if is_service_enabled cinder; then
        iniset $NOVA_CONF DEFAULT volume_api_class "nova.volume.cinder.API"
    fi
@@ -453,32 +464,38 @@ function create_nova_conf() {
    fi

    if is_service_enabled n-novnc || is_service_enabled n-xvnc; then
-      # Address on which instance vncservers will listen on compute hosts.
-      # For multi-host, this should be the management ip of the compute host.
-      VNCSERVER_LISTEN=${VNCSERVER_LISTEN=127.0.0.1}
-      VNCSERVER_PROXYCLIENT_ADDRESS=${VNCSERVER_PROXYCLIENT_ADDRESS=127.0.0.1}
-      iniset $NOVA_CONF DEFAULT vnc_enabled true
-      iniset $NOVA_CONF DEFAULT vncserver_listen "$VNCSERVER_LISTEN"
-      iniset $NOVA_CONF DEFAULT vncserver_proxyclient_address "$VNCSERVER_PROXYCLIENT_ADDRESS"
+        # Address on which instance vncservers will listen on compute hosts.
+        # For multi-host, this should be the management ip of the compute host.
+        VNCSERVER_LISTEN=${VNCSERVER_LISTEN=127.0.0.1}
+        VNCSERVER_PROXYCLIENT_ADDRESS=${VNCSERVER_PROXYCLIENT_ADDRESS=127.0.0.1}
+        iniset $NOVA_CONF DEFAULT vnc_enabled true
+        iniset $NOVA_CONF DEFAULT vncserver_listen "$VNCSERVER_LISTEN"
+        iniset $NOVA_CONF DEFAULT vncserver_proxyclient_address "$VNCSERVER_PROXYCLIENT_ADDRESS"
    else
-      iniset $NOVA_CONF DEFAULT vnc_enabled false
+        iniset $NOVA_CONF DEFAULT vnc_enabled false
    fi

    if is_service_enabled n-spice; then
-      # Address on which instance spiceservers will listen on compute hosts.
-      # For multi-host, this should be the management ip of the compute host.
-      SPICESERVER_PROXYCLIENT_ADDRESS=${SPICESERVER_PROXYCLIENT_ADDRESS=127.0.0.1}
-      SPICESERVER_LISTEN=${SPICESERVER_LISTEN=127.0.0.1}
-      iniset $NOVA_CONF spice enabled true
-      iniset $NOVA_CONF spice server_listen "$SPICESERVER_LISTEN"
-      iniset $NOVA_CONF spice server_proxyclient_address "$SPICESERVER_PROXYCLIENT_ADDRESS"
+        # Address on which instance spiceservers will listen on compute hosts.
+        # For multi-host, this should be the management ip of the compute host.
+        SPICESERVER_PROXYCLIENT_ADDRESS=${SPICESERVER_PROXYCLIENT_ADDRESS=127.0.0.1}
+        SPICESERVER_LISTEN=${SPICESERVER_LISTEN=127.0.0.1}
+        iniset $NOVA_CONF spice enabled true
+        iniset $NOVA_CONF spice server_listen "$SPICESERVER_LISTEN"
+        iniset $NOVA_CONF spice server_proxyclient_address "$SPICESERVER_PROXYCLIENT_ADDRESS"
    else
-      iniset $NOVA_CONF spice enabled false
+        iniset $NOVA_CONF spice enabled false
    fi

    iniset $NOVA_CONF DEFAULT ec2_dmz_host "$EC2_DMZ_HOST"
    iniset_rpc_backend nova $NOVA_CONF DEFAULT
    iniset $NOVA_CONF DEFAULT glance_api_servers "$GLANCE_HOSTPORT"
+
+    if [[ "$VIRT_DRIVER" = 'libvirt' ]]; then
+        # File injection is being disabled by default in the near future -
+        # disable it here for now to avoid surprises later.
+        iniset $NOVA_CONF libvirt inject_partition '-2'
+    fi
 }

 function init_nova_cells() {
@@ -590,7 +607,7 @@ function start_nova_api() {
    screen_it n-api "cd $NOVA_DIR && $NOVA_BIN_DIR/nova-api"
    echo "Waiting for nova-api to start..."
    if ! wait_for_service $SERVICE_TIMEOUT http://$SERVICE_HOST:$service_port; then
-      die $LINENO "nova-api did not start"
+        die $LINENO "nova-api did not start"
    fi

    # Start proxies if enabled
@@ -618,10 +635,9 @@ function start_nova() {
        # Use 'sg' to execute nova-compute as a member of the **$LIBVIRT_GROUP** group.
        screen_it n-cpu "cd $NOVA_DIR && sg $LIBVIRT_GROUP '$NOVA_BIN_DIR/nova-compute --config-file $NOVA_CONF_BOTTOM'"
    elif [[ "$VIRT_DRIVER" = 'fake' ]]; then
-       for i in `seq 1 $NUMBER_FAKE_NOVA_COMPUTE`
-       do
-           screen_it n-cpu "cd $NOVA_DIR && $NOVA_BIN_DIR/nova-compute --config-file $NOVA_CONF_BOTTOM"
-       done
+        for i in `seq 1 $NUMBER_FAKE_NOVA_COMPUTE`; do
+            screen_it n-cpu "cd $NOVA_DIR && $NOVA_BIN_DIR/nova-compute --config-file $NOVA_CONF_BOTTOM"
+        done
    else
        if is_service_enabled n-cpu && [[ -r $NOVA_PLUGINS/hypervisor-$VIRT_DRIVER ]]; then
            start_nova_hypervisor
@@ -644,19 +660,27 @@ function start_nova() {
        screen_it n-obj "cd $NOVA_DIR && $NOVA_BIN_DIR/nova-objectstore"
 }

-# stop_nova() - Stop running processes (non-screen)
-function stop_nova() {
-    # Kill the nova screen windows
-    # Some services are listed here twice since more than one instance
-    # of a service may be running in certain configs.
-    for serv in n-api n-cpu n-crt n-net n-sch n-novnc n-xvnc n-cauth n-spice n-cond n-cond n-cell n-cell n-api-meta; do
-        screen -S $SCREEN_NAME -p $serv -X kill
-    done
+function stop_nova_compute {
+    screen_stop n-cpu
    if is_service_enabled n-cpu && [[ -r $NOVA_PLUGINS/hypervisor-$VIRT_DRIVER ]]; then
        stop_nova_hypervisor
    fi
 }

+function stop_nova_rest {
+    # Kill the nova screen windows
+    # Some services are listed here twice since more than one instance
+    # of a service may be running in certain configs.
+    for serv in n-api n-crt n-net n-sch n-novnc n-xvnc n-cauth n-spice n-cond n-cond n-cell n-cell n-api-meta; do
+        screen_stop $serv
+    done
+}
+
+# stop_nova() - Stop running processes (non-screen)
+function stop_nova() {
+    stop_nova_rest
+    stop_nova_compute
+}

 # Restore xtrace
 $XTRACE
@@ -61,8 +61,8 @@ function configure_nova_hypervisor() {

    # Define extra baremetal nova conf flags by defining the array ``EXTRA_BAREMETAL_OPTS``.
    for I in "${EXTRA_BAREMETAL_OPTS[@]}"; do
-       # Attempt to convert flags to options
-       iniset $NOVA_CONF baremetal ${I/=/ }
+        # Attempt to convert flags to options
+        iniset $NOVA_CONF baremetal ${I/=/ }
    done
 }

@@ -22,6 +22,8 @@ set +o xtrace
 # Defaults
 # --------

+# File injection is disabled by default in Nova.  This will turn it back on.
+ENABLE_FILE_INJECTION=${ENABLE_FILE_INJECTION:-False}

 # Entry Points
 # ------------
@@ -82,10 +84,10 @@ EOF"
            sudo mkdir -p $rules_dir
            sudo bash -c "cat <<EOF > $rules_dir/50-libvirt-$STACK_USER.rules
 polkit.addRule(function(action, subject) {
-     if (action.id == 'org.libvirt.unix.manage' &&
-         subject.user == '"$STACK_USER"') {
-         return polkit.Result.YES;
-     }
+    if (action.id == 'org.libvirt.unix.manage' &&
+        subject.user == '"$STACK_USER"') {
+        return polkit.Result.YES;
+    }
 });
 EOF"
            unset rules_dir
@@ -114,6 +116,10 @@ EOF"
    if is_arch "ppc64"; then
        iniset $NOVA_CONF DEFAULT vnc_enabled "false"
    fi
+    if [[ "$ENABLE_FILE_INJECTION" = "False" ]] ; then
+        # Disable file injection by default
+        iniset $NOVA_CONF DEFAULT libvirt_inject_partition '-2'
+    fi
 }

 # install_nova_hypervisor() - Install external components
@@ -27,15 +27,13 @@ OSLOMSG_DIR=$DEST/oslo.messaging

 # install_oslo() - Collect source and prepare
 function install_oslo() {
-    # TODO(sdague): remove this once we get to Icehouse, this just makes
-    # for a smoother transition of existing users.
    cleanup_oslo

    git_clone $OSLOCFG_REPO $OSLOCFG_DIR $OSLOCFG_BRANCH
-    setup_develop $OSLOCFG_DIR
+    setup_install $OSLOCFG_DIR

    git_clone $OSLOMSG_REPO $OSLOMSG_DIR $OSLOMSG_BRANCH
-    setup_develop $OSLOMSG_DIR
+    setup_install $OSLOMSG_DIR
 }

 # cleanup_oslo() - purge possibly old versions of oslo
@@ -102,9 +102,9 @@ function install_rpc_backend() {
        if is_fedora; then
            install_package qpid-cpp-server
            if [[ $DISTRO =~ (rhel6) ]]; then
-               # RHEL6 leaves "auth=yes" in /etc/qpidd.conf, it needs to
-               # be no or you get GSS authentication errors as it
-               # attempts to default to this.
+                # RHEL6 leaves "auth=yes" in /etc/qpidd.conf, it needs to
+                # be no or you get GSS authentication errors as it
+                # attempts to default to this.
                sudo sed -i.bak 's/^auth=yes$/auth=no/' /etc/qpidd.conf
            fi
        elif is_ubuntu; then
@@ -0,0 +1,68 @@
+# lib/stackforge
+#
+# Functions to install stackforge libraries that we depend on so
+# that we can try their git versions during devstack gate.
+#
+# This is appropriate for python libraries that release to pypi and are
+# expected to be used beyond OpenStack like, but are requirements
+# for core services in global-requirements.
+#
+#     * wsme
+#     * pecan
+#
+# This is not appropriate for stackforge projects which are early stage
+# OpenStack tools
+
+# Dependencies:
+# ``functions`` file
+
+# ``stack.sh`` calls the entry points in this order:
+#
+# install_stackforge
+
+# Save trace setting
+XTRACE=$(set +o | grep xtrace)
+set +o xtrace
+
+
+# Defaults
+# --------
+WSME_DIR=$DEST/wsme
+PECAN_DIR=$DEST/pecan
+
+# Entry Points
+# ------------
+
+# install_stackforge() - Collect source and prepare
+function install_stackforge {
+    # TODO(sdague): remove this once we get to Icehouse, this just makes
+    # for a smoother transition of existing users.
+    cleanup_stackforge
+
+    git_clone $WSME_REPO $WSME_DIR $WSME_BRANCH
+    setup_package $WSME_DIR
+
+    git_clone $PECAN_REPO $PECAN_DIR $PECAN_BRANCH
+    setup_package $PECAN_DIR
+}
+
+# cleanup_stackforge() - purge possibly old versions of stackforge libraries
+function cleanup_stackforge {
+    # this means we've got an old version installed, lets get rid of it
+    # otherwise python hates itself
+    for lib in wsme pecan; do
+        if ! python -c "import $lib" 2>/dev/null; then
+            echo "Found old $lib... removing to ensure consistency"
+            local PIP_CMD=$(get_pip_command)
+            pip_install $lib
+            sudo $PIP_CMD uninstall -y $lib
+        fi
+    done
+}
+
+# Restore xtrace
+$XTRACE
+
+# Local variables:
+# mode: shell-script
+# End:
@@ -104,17 +104,17 @@ ACCOUNT_PORT_BASE=${ACCOUNT_PORT_BASE:-6012}

 # cleanup_swift() - Remove residual data files
 function cleanup_swift() {
-   rm -f ${SWIFT_CONF_DIR}{*.builder,*.ring.gz,backups/*.builder,backups/*.ring.gz}
-   if egrep -q ${SWIFT_DATA_DIR}/drives/sdb1 /proc/mounts; then
-      sudo umount ${SWIFT_DATA_DIR}/drives/sdb1
-   fi
-   if [[ -e ${SWIFT_DISK_IMAGE} ]]; then
-      rm ${SWIFT_DISK_IMAGE}
-   fi
-   rm -rf ${SWIFT_DATA_DIR}/run/
-   if is_apache_enabled_service swift; then
-       _cleanup_swift_apache_wsgi
-   fi
+    rm -f ${SWIFT_CONF_DIR}{*.builder,*.ring.gz,backups/*.builder,backups/*.ring.gz}
+    if egrep -q ${SWIFT_DATA_DIR}/drives/sdb1 /proc/mounts; then
+        sudo umount ${SWIFT_DATA_DIR}/drives/sdb1
+    fi
+    if [[ -e ${SWIFT_DISK_IMAGE} ]]; then
+        rm ${SWIFT_DISK_IMAGE}
+    fi
+    rm -rf ${SWIFT_DATA_DIR}/run/
+    if is_apache_enabled_service swift; then
+        _cleanup_swift_apache_wsgi
+    fi
 }

 # _cleanup_swift_apache_wsgi() - Remove wsgi files, disable and remove apache vhost file
@@ -192,7 +192,7 @@ function _config_swift_apache_wsgi() {

        sudo cp ${SWIFT_DIR}/examples/apache2/account-server.template ${apache_vhost_dir}/account-server-${node_number}
        sudo sed -e "
-             /^#/d;/^$/d;
+            /^#/d;/^$/d;
            s/%PORT%/$account_port/g;
            s/%SERVICENAME%/account-server-${node_number}/g;
            s/%APACHE_NAME%/${APACHE_NAME}/g;
@@ -202,7 +202,7 @@ function _config_swift_apache_wsgi() {

        sudo cp ${SWIFT_DIR}/examples/wsgi/account-server.wsgi.template ${SWIFT_APACHE_WSGI_DIR}/account-server-${node_number}.wsgi
        sudo sed -e "
-             /^#/d;/^$/d;
+            /^#/d;/^$/d;
            s/%SERVICECONF%/account-server\/${node_number}.conf/g;
        " -i ${SWIFT_APACHE_WSGI_DIR}/account-server-${node_number}.wsgi
    done
@@ -260,6 +260,15 @@ function configure_swift() {
    iniuncomment ${SWIFT_CONFIG_PROXY_SERVER} DEFAULT bind_port
    iniset ${SWIFT_CONFIG_PROXY_SERVER} DEFAULT bind_port ${SWIFT_DEFAULT_BIND_PORT:-8080}

+    # Devstack is commonly run in a small slow environment, so bump the
+    # timeouts up.
+    # node_timeout is how long between read operations a node takes to
+    # respond to the proxy server
+    # conn_timeout is all about how long it takes a connect() system call to
+    # return
+    iniset ${SWIFT_CONFIG_PROXY_SERVER} app:proxy-server node_timeout 120
+    iniset ${SWIFT_CONFIG_PROXY_SERVER} app:proxy-server conn_timeout 20
+
    # Configure Ceilometer
    if is_service_enabled ceilometer; then
        iniset ${SWIFT_CONFIG_PROXY_SERVER} filter:ceilometer use "egg:ceilometer#swift"
@@ -435,7 +444,7 @@ function create_swift_disk() {
    truncate -s ${SWIFT_LOOPBACK_DISK_SIZE} ${SWIFT_DISK_IMAGE}

    # Make a fresh XFS filesystem
-    mkfs.xfs -f -i size=1024  ${SWIFT_DISK_IMAGE}
+    /sbin/mkfs.xfs -f -i size=1024  ${SWIFT_DISK_IMAGE}

    # Mount the disk with mount options to make it as efficient as possible
    mkdir -p ${SWIFT_DATA_DIR}/drives/sdb1
@@ -577,26 +586,26 @@ function start_swift() {
        return 0
    fi

-   # By default with only one replica we are launching the proxy,
-   # container, account and object server in screen in foreground and
-   # other services in background. If we have SWIFT_REPLICAS set to something
-   # greater than one we first spawn all the swift services then kill the proxy
-   # service so we can run it in foreground in screen.  ``swift-init ...
-   # {stop|restart}`` exits with '1' if no servers are running, ignore it just
-   # in case
-   swift-init --run-dir=${SWIFT_DATA_DIR}/run all restart || true
-   if [[ ${SWIFT_REPLICAS} == 1 ]]; then
+    # By default with only one replica we are launching the proxy,
+    # container, account and object server in screen in foreground and
+    # other services in background. If we have SWIFT_REPLICAS set to something
+    # greater than one we first spawn all the swift services then kill the proxy
+    # service so we can run it in foreground in screen.  ``swift-init ...
+    # {stop|restart}`` exits with '1' if no servers are running, ignore it just
+    # in case
+    swift-init --run-dir=${SWIFT_DATA_DIR}/run all restart || true
+    if [[ ${SWIFT_REPLICAS} == 1 ]]; then
        todo="object container account"
-   fi
-   for type in proxy ${todo}; do
-       swift-init --run-dir=${SWIFT_DATA_DIR}/run ${type} stop || true
-   done
-   screen_it s-proxy "cd $SWIFT_DIR && $SWIFT_DIR/bin/swift-proxy-server ${SWIFT_CONF_DIR}/proxy-server.conf -v"
-   if [[ ${SWIFT_REPLICAS} == 1 ]]; then
-       for type in object container account; do
-           screen_it s-${type} "cd $SWIFT_DIR && $SWIFT_DIR/bin/swift-${type}-server ${SWIFT_CONF_DIR}/${type}-server/1.conf -v"
-       done
-   fi
+    fi
+    for type in proxy ${todo}; do
+        swift-init --run-dir=${SWIFT_DATA_DIR}/run ${type} stop || true
+    done
+    screen_it s-proxy "cd $SWIFT_DIR && $SWIFT_DIR/bin/swift-proxy-server ${SWIFT_CONF_DIR}/proxy-server.conf -v"
+    if [[ ${SWIFT_REPLICAS} == 1 ]]; then
+        for type in object container account; do
+            screen_it s-${type} "cd $SWIFT_DIR && $SWIFT_DIR/bin/swift-${type}-server ${SWIFT_CONF_DIR}/${type}-server/1.conf -v"
+        done
+    fi
 }

 # stop_swift() - Stop running processes (non-screen)
@@ -610,8 +619,13 @@ function stop_swift() {
    if type -p swift-init >/dev/null; then
        swift-init --run-dir=${SWIFT_DATA_DIR}/run all stop || true
    fi
-    # Dump the proxy server
-    sudo pkill -f swift-proxy-server
+    # Dump all of the servers
+    # Maintain the iteration as screen_stop() has some desirable side-effects
+    for type in proxy object container account; do
+        screen_stop s-${type}
+    done
+    # Blast out any stragglers
+    pkill -f swift-
 }

 # Restore xtrace
@@ -14,6 +14,7 @@
 # - ``PUBLIC_NETWORK_NAME``
 # - ``Q_USE_NAMESPACE``
 # - ``Q_ROUTER_NAME``
+# - ``Q_L3_ENABLED``
 # - ``VIRT_DRIVER``
 # - ``LIBVIRT_TYPE``
 # - ``KEYSTONE_SERVICE_PROTOCOL``, ``KEYSTONE_SERVICE_HOST`` from lib/keystone
@@ -130,8 +131,12 @@ function configure_tempest() {

    password=${ADMIN_PASSWORD:-secrete}

-    # See files/keystone_data.sh where alt_demo user
-    # and tenant are set up...
+    # See files/keystone_data.sh and stack.sh where admin, demo and alt_demo
+    # user and tenant are set up...
+    ADMIN_USERNAME=${ADMIN_USERNAME:-admin}
+    ADMIN_TENANT_NAME=${ADMIN_TENANT_NAME:-admin}
+    TEMPEST_USERNAME=${TEMPEST_USERNAME:-demo}
+    TEMPEST_TENANT_NAME=${TEMPEST_TENANT_NAME:-demo}
    ALT_USERNAME=${ALT_USERNAME:-alt_demo}
    ALT_TENANT_NAME=${ALT_TENANT_NAME:-alt_demo}

@@ -186,14 +191,14 @@ function configure_tempest() {
        tenant_networks_reachable=true
    fi

-    if is_service_enabled q-l3; then
+    if [ "$Q_L3_ENABLED" = "True" ]; then
        public_network_id=$(neutron net-list | grep $PUBLIC_NETWORK_NAME | \
            awk '{print $2}')
        if [ "$Q_USE_NAMESPACE" == "False" ]; then
            # If namespaces are disabled, devstack will create a single
            # public router that tempest should be configured to use.
            public_router_id=$(neutron router-list | awk "/ $Q_ROUTER_NAME / \
-               { print \$2 }")
+                { print \$2 }")
        fi
    fi

@@ -215,11 +220,16 @@ function configure_tempest() {

    # Identity
    iniset $TEMPEST_CONF identity uri "$KEYSTONE_SERVICE_PROTOCOL://$KEYSTONE_SERVICE_HOST:5000/v2.0/"
+    iniset $TEMPEST_CONF identity uri_v3 "$KEYSTONE_SERVICE_PROTOCOL://$KEYSTONE_SERVICE_HOST:5000/v3/"
+    iniset $TEMPEST_CONF identity username $TEMPEST_USERNAME
    iniset $TEMPEST_CONF identity password "$password"
+    iniset $TEMPEST_CONF identity tenant_name $TEMPEST_TENANT_NAME
    iniset $TEMPEST_CONF identity alt_username $ALT_USERNAME
    iniset $TEMPEST_CONF identity alt_password "$password"
    iniset $TEMPEST_CONF identity alt_tenant_name $ALT_TENANT_NAME
+    iniset $TEMPEST_CONF identity admin_username $ADMIN_USERNAME
    iniset $TEMPEST_CONF identity admin_password "$password"
+    iniset $TEMPEST_CONF identity admin_tenant_name $ADMIN_TENANT_NAME

    # Image
    # for the gate we want to be able to override this variable so we aren't
@@ -250,7 +260,9 @@ function configure_tempest() {
    iniset $TEMPEST_CONF compute use_block_migration_for_live_migration ${USE_BLOCK_MIGRATION_FOR_LIVE_MIGRATION:-False}

    # Compute admin
+    iniset $TEMPEST_CONF "compute-admin" username $ADMIN_USERNAME
    iniset $TEMPEST_CONF "compute-admin" password "$password" # DEPRECATED
+    iniset $TEMPEST_CONF "compute-admin" tenant_name $ADMIN_TENANT_NAME

    iniset $TEMPEST_CONF network api_version 2.0
    iniset $TEMPEST_CONF network tenant_networks_reachable "$tenant_networks_reachable"
@@ -328,15 +340,15 @@ function init_tempest() {
    local disk_image="$image_dir/${base_image_name}-blank.img"
    # if the cirros uec downloaded and the system is uec capable
    if [ -f "$kernel" -a -f "$ramdisk" -a -f "$disk_image" -a  "$VIRT_DRIVER" != "openvz" \
-         -a \( "$LIBVIRT_TYPE" != "lxc" -o "$VIRT_DRIVER" != "libvirt" \) ]; then
-       echo "Prepare aki/ari/ami Images"
-       ( #new namespace
-           # tenant:demo ; user: demo
-           source $TOP_DIR/accrc/demo/demo
-           euca-bundle-image -i "$kernel" --kernel true -d "$BOTO_MATERIALS_PATH"
-           euca-bundle-image -i "$ramdisk" --ramdisk true -d "$BOTO_MATERIALS_PATH"
-           euca-bundle-image -i "$disk_image" -d "$BOTO_MATERIALS_PATH"
-       ) 2>&1 </dev/null | cat
+        -a \( "$LIBVIRT_TYPE" != "lxc" -o "$VIRT_DRIVER" != "libvirt" \) ]; then
+        echo "Prepare aki/ari/ami Images"
+        ( #new namespace
+            # tenant:demo ; user: demo
+            source $TOP_DIR/accrc/demo/demo
+            euca-bundle-image -r x86_64 -i "$kernel" --kernel true -d "$BOTO_MATERIALS_PATH"
+            euca-bundle-image -r x86_64 -i "$ramdisk" --ramdisk true -d "$BOTO_MATERIALS_PATH"
+            euca-bundle-image -r x86_64 -i "$disk_image" -d "$BOTO_MATERIALS_PATH"
+        ) 2>&1 </dev/null | cat
    else
        echo "Boto materials are not prepared"
    fi
@@ -45,14 +45,15 @@ create_trove_accounts() {
    SERVICE_ROLE=$(keystone role-list | awk "/ admin / { print \$2 }")

    if [[ "$ENABLED_SERVICES" =~ "trove" ]]; then
-        TROVE_USER=$(keystone user-create --name=trove \
-                                                  --pass="$SERVICE_PASSWORD" \
-                                                  --tenant_id $SERVICE_TENANT \
-                                                  --email=trove@example.com \
-                                                  | grep " id " | get_field 2)
+        TROVE_USER=$(keystone user-create \
+            --name=trove \
+            --pass="$SERVICE_PASSWORD" \
+            --tenant_id $SERVICE_TENANT \
+            --email=trove@example.com \
+            | grep " id " | get_field 2)
        keystone user-role-add --tenant-id $SERVICE_TENANT \
-                               --user-id $TROVE_USER \
-                               --role-id $SERVICE_ROLE
+            --user-id $TROVE_USER \
+            --role-id $SERVICE_ROLE
        if [[ "$KEYSTONE_CATALOG_BACKEND" = 'sql' ]]; then
            TROVE_SERVICE=$(keystone service-create \
                --name=trove \
@@ -173,7 +174,7 @@ function start_trove() {
 function stop_trove() {
    # Kill the trove screen windows
    for serv in tr-api tr-tmgr tr-cond; do
-        screen -S $SCREEN_NAME -p $serv -X kill
+        screen_stop $serv
    done
 }

@@ -53,7 +53,7 @@ if [[ -r $TOP_DIR/local.conf ]]; then
            if [[ -r $TOP_DIR/localrc ]]; then
                warn $LINENO "localrc and local.conf:[[local]] both exist, using localrc"
            else
-                echo "# Generated file, do not exit" >$TOP_DIR/.localrc.auto
+                echo "# Generated file, do not edit" >$TOP_DIR/.localrc.auto
                get_meta_section $TOP_DIR/local.conf local $lfile >>$TOP_DIR/.localrc.auto
            fi
        fi
@@ -158,6 +158,11 @@ VERBOSE=$(trueorfalse True $VERBOSE)
 # Additional repos
 # ================

+# For debian/ubuntu make apt attempt to retry network ops on it's own
+if is_ubuntu; then
+    echo 'APT::Acquire::Retries "20";' | sudo tee /etc/apt/apt.conf.d/80retry
+fi
+
 # Some distros need to add repos beyond the defaults provided by the vendor
 # to pick up required packages.

@@ -497,23 +502,15 @@ if [[ -n "$LOGFILE" ]]; then
    # Copy stdout to fd 3
    exec 3>&1
    if [[ "$VERBOSE" == "True" ]]; then
-        # Redirect stdout/stderr to tee to write the log file
-        exec 1> >( awk '
-                {
-                    cmd ="date +\"%Y-%m-%d %H:%M:%S \""
-                    cmd | getline now
-                    close("date +\"%Y-%m-%d %H:%M:%S \"")
-                    sub(/^/, now)
-                    print
-                    fflush()
-                }' | tee "${LOGFILE}" ) 2>&1
-        # Set up a second fd for output
-        exec 6> >( tee "${SUMFILE}" )
+        # Set fd 1 and 2 to write the log file
+        exec 1> >( $TOP_DIR/tools/outfilter.py -v -o "${LOGFILE}" ) 2>&1
+        # Set fd 6 to summary log file
+        exec 6> >( $TOP_DIR/tools/outfilter.py -o "${SUMFILE}" )
    else
        # Set fd 1 and 2 to primary logfile
-        exec 1> "${LOGFILE}" 2>&1
+        exec 1> >( $TOP_DIR/tools/outfilter.py -o "${LOGFILE}" ) 2>&1
        # Set fd 6 to summary logfile and stdout
-        exec 6> >( tee "${SUMFILE}" >&3 )
+        exec 6> >( $TOP_DIR/tools/outfilter.py -v -o "${SUMFILE}" >&3 )
    fi

    echo_summary "stack.sh log $LOGFILE"
@@ -529,7 +526,7 @@ else
        exec 1>/dev/null 2>&1
    fi
    # Always send summary fd to original stdout
-    exec 6>&3
+    exec 6> >( $TOP_DIR/tools/outfilter.py -v >&3 )
 fi

 # Set up logging of screen windows
@@ -697,6 +694,8 @@ if is_service_enabled n-spice; then
 fi

 if is_service_enabled horizon; then
+    # django openstack_auth
+    install_django_openstack_auth
    # dashboard
    install_horizon
    configure_horizon
@@ -1015,7 +1014,7 @@ if is_service_enabled nova && is_baremetal; then
    prepare_baremetal_toolchain
    configure_baremetal_nova_dirs
    if [[ "$BM_USE_FAKE_ENV" = "True" ]]; then
-       create_fake_baremetal_env
+        create_fake_baremetal_env
    fi
 fi

@@ -1074,6 +1073,12 @@ if is_service_enabled key && is_service_enabled swift3 && is_service_enabled nov
    iniset $NOVA_CONF DEFAULT s3_affix_tenant "True"
 fi

+# Create a randomized default value for the keymgr's fixed_key
+if is_service_enabled nova; then
+    FIXED_KEY=$(cat /dev/urandom | tr -cd 'A-F0-9' | head -c 64)
+    iniset $NOVA_CONF keymgr fixed_key "$FIXED_KEY"
+fi
+
 if is_service_enabled zeromq; then
    echo_summary "Starting zermomq receiver"
    screen_it zeromq "cd $NOVA_DIR && $NOVA_BIN_DIR/nova-rpc-zmq-receiver"
@@ -1087,10 +1092,7 @@ fi

 if is_service_enabled q-svc; then
    echo_summary "Starting Neutron"
-
    start_neutron_service_and_check
-    create_neutron_initial_network
-    setup_neutron_debug
 elif is_service_enabled $DATABASE_BACKENDS && is_service_enabled n-net; then
    NM_CONF=${NOVA_CONF}
    if is_service_enabled n-cell; then
@@ -1110,6 +1112,12 @@ fi
 if is_service_enabled neutron; then
    start_neutron_agents
 fi
+# Once neutron agents are started setup initial network elements
+if is_service_enabled q-svc; then
+    echo_summary "Creating initial neutron network elements"
+    create_neutron_initial_network
+    setup_neutron_debug
+fi
 if is_service_enabled nova; then
    echo_summary "Starting Nova"
    start_nova
@@ -1176,26 +1184,26 @@ if is_service_enabled g-reg; then
    TOKEN=$(keystone token-get | grep ' id ' | get_field 2)

    if is_baremetal; then
-       echo_summary "Creating and uploading baremetal images"
+        echo_summary "Creating and uploading baremetal images"

-       # build and upload separate deploy kernel & ramdisk
-       upload_baremetal_deploy $TOKEN
+        # build and upload separate deploy kernel & ramdisk
+        upload_baremetal_deploy $TOKEN

-       # upload images, separating out the kernel & ramdisk for PXE boot
-       for image_url in ${IMAGE_URLS//,/ }; do
-           upload_baremetal_image $image_url $TOKEN
-       done
+        # upload images, separating out the kernel & ramdisk for PXE boot
+        for image_url in ${IMAGE_URLS//,/ }; do
+            upload_baremetal_image $image_url $TOKEN
+        done
    else
-       echo_summary "Uploading images"
+        echo_summary "Uploading images"

-       # Option to upload legacy ami-tty, which works with xenserver
-       if [[ -n "$UPLOAD_LEGACY_TTY" ]]; then
-           IMAGE_URLS="${IMAGE_URLS:+${IMAGE_URLS},}https://github.com/downloads/citrix-openstack/warehouse/tty.tgz"
-       fi
+        # Option to upload legacy ami-tty, which works with xenserver
+        if [[ -n "$UPLOAD_LEGACY_TTY" ]]; then
+            IMAGE_URLS="${IMAGE_URLS:+${IMAGE_URLS},}https://github.com/downloads/citrix-openstack/warehouse/tty.tgz"
+        fi

-       for image_url in ${IMAGE_URLS//,/ }; do
-           upload_image $image_url $TOKEN
-       done
+        for image_url in ${IMAGE_URLS//,/ }; do
+            upload_image $image_url $TOKEN
+        done
    fi
 fi

@@ -1207,7 +1215,7 @@ fi
 if is_service_enabled nova && is_baremetal; then
    # create special flavor for baremetal if we know what images to associate
    [[ -n "$BM_DEPLOY_KERNEL_ID" ]] && [[ -n "$BM_DEPLOY_RAMDISK_ID" ]] && \
-       create_baremetal_flavor $BM_DEPLOY_KERNEL_ID $BM_DEPLOY_RAMDISK_ID
+        create_baremetal_flavor $BM_DEPLOY_KERNEL_ID $BM_DEPLOY_RAMDISK_ID

    # otherwise user can manually add it later by calling nova-baremetal-manage
    [[ -n "$BM_FIRST_MAC" ]] && add_baremetal_node
@@ -1229,7 +1237,7 @@ fi
 CURRENT_RUN_TIME=$(date "+$TIMESTAMP_FORMAT")
 echo "# $CURRENT_RUN_TIME" >$TOP_DIR/.stackenv
 for i in BASE_SQL_CONN ENABLED_SERVICES HOST_IP LOGFILE \
-  SERVICE_HOST SERVICE_PROTOCOL STACK_USER TLS_IP; do
+    SERVICE_HOST SERVICE_PROTOCOL STACK_USER TLS_IP; do
    echo $i=${!i} >>$TOP_DIR/.stackenv
 done

@@ -9,6 +9,9 @@ DEST=/opt/stack
 # Destination for working data
 DATA_DIR=${DEST}/data

+# Destination for status files
+SERVICE_DIR=${DEST}/status
+
 # Determine stack user
 if [[ $EUID -eq 0 ]]; then
    STACK_USER=stack
@@ -66,7 +69,7 @@ GIT_BASE=${GIT_BASE:-https://github.com}

 # metering service
 CEILOMETER_REPO=${CEILOMETER_REPO:-${GIT_BASE}/openstack/ceilometer.git}
-CEILOMETER_BRANCH=${CEILOMETER_BRANCH:-master}
+CEILOMETER_BRANCH=${CEILOMETER_BRANCH:-stable/havana}

 # ceilometer client library
 CEILOMETERCLIENT_REPO=${CEILOMETERCLIENT_REPO:-${GIT_BASE}/openstack/python-ceilometerclient.git}
@@ -74,7 +77,7 @@ CEILOMETERCLIENT_BRANCH=${CEILOMETERCLIENT_BRANCH:-master}

 # volume service
 CINDER_REPO=${CINDER_REPO:-${GIT_BASE}/openstack/cinder.git}
-CINDER_BRANCH=${CINDER_BRANCH:-master}
+CINDER_BRANCH=${CINDER_BRANCH:-stable/havana}

 # volume client
 CINDERCLIENT_REPO=${CINDERCLIENT_REPO:-${GIT_BASE}/openstack/python-cinderclient.git}
@@ -82,7 +85,7 @@ CINDERCLIENT_BRANCH=${CINDERCLIENT_BRANCH:-master}

 # image catalog service
 GLANCE_REPO=${GLANCE_REPO:-${GIT_BASE}/openstack/glance.git}
-GLANCE_BRANCH=${GLANCE_BRANCH:-master}
+GLANCE_BRANCH=${GLANCE_BRANCH:-stable/havana}

 # python glance client library
 GLANCECLIENT_REPO=${GLANCECLIENT_REPO:-${GIT_BASE}/openstack/python-glanceclient.git}
@@ -90,7 +93,7 @@ GLANCECLIENT_BRANCH=${GLANCECLIENT_BRANCH:-master}

 # heat service
 HEAT_REPO=${HEAT_REPO:-${GIT_BASE}/openstack/heat.git}
-HEAT_BRANCH=${HEAT_BRANCH:-master}
+HEAT_BRANCH=${HEAT_BRANCH:-stable/havana}

 # python heat client library
 HEATCLIENT_REPO=${HEATCLIENT_REPO:-${GIT_BASE}/openstack/python-heatclient.git}
@@ -98,7 +101,11 @@ HEATCLIENT_BRANCH=${HEATCLIENT_BRANCH:-master}

 # django powered web control panel for openstack
 HORIZON_REPO=${HORIZON_REPO:-${GIT_BASE}/openstack/horizon.git}
-HORIZON_BRANCH=${HORIZON_BRANCH:-master}
+HORIZON_BRANCH=${HORIZON_BRANCH:-stable/havana}
+
+# django openstack_auth library
+HORIZONAUTH_REPO=${HORIZONAUTH_REPO:-${GIT_BASE}/openstack/django_openstack_auth.git}
+HORIZONAUTH_BRANCH=${HORIZONAUTH_BRANCH:-master}

 # baremetal provisionint service
 IRONIC_REPO=${IRONIC_REPO:-${GIT_BASE}/openstack/ironic.git}
@@ -106,7 +113,7 @@ IRONIC_BRANCH=${IRONIC_BRANCH:-master}

 # unified auth system (manages accounts/tokens)
 KEYSTONE_REPO=${KEYSTONE_REPO:-${GIT_BASE}/openstack/keystone.git}
-KEYSTONE_BRANCH=${KEYSTONE_BRANCH:-master}
+KEYSTONE_BRANCH=${KEYSTONE_BRANCH:-stable/havana}

 # python keystone client library to nova that horizon uses
 KEYSTONECLIENT_REPO=${KEYSTONECLIENT_REPO:-${GIT_BASE}/openstack/python-keystoneclient.git}
@@ -114,7 +121,7 @@ KEYSTONECLIENT_BRANCH=${KEYSTONECLIENT_BRANCH:-master}

 # compute service
 NOVA_REPO=${NOVA_REPO:-${GIT_BASE}/openstack/nova.git}
-NOVA_BRANCH=${NOVA_BRANCH:-master}
+NOVA_BRANCH=${NOVA_BRANCH:-stable/havana}

 # python client library to nova that horizon (and others) use
 NOVACLIENT_REPO=${NOVACLIENT_REPO:-${GIT_BASE}/openstack/python-novaclient.git}
@@ -126,7 +133,7 @@ OPENSTACKCLIENT_BRANCH=${OPENSTACKCLIENT_BRANCH:-master}

 # oslo.config
 OSLOCFG_REPO=${OSLOCFG_REPO:-${GIT_BASE}/openstack/oslo.config.git}
-OSLOCFG_BRANCH=${OSLOCFG_BRANCH:-master}
+OSLOCFG_BRANCH=${OSLOCFG_BRANCH:-stable/havana}

 # oslo.messaging
 OSLOMSG_REPO=${OSLOMSG_REPO:-${GIT_BASE}/openstack/oslo.messaging.git}
@@ -138,7 +145,7 @@ PBR_BRANCH=${PBR_BRANCH:-master}

 # neutron service
 NEUTRON_REPO=${NEUTRON_REPO:-${GIT_BASE}/openstack/neutron.git}
-NEUTRON_BRANCH=${NEUTRON_BRANCH:-master}
+NEUTRON_BRANCH=${NEUTRON_BRANCH:-stable/havana}

 # neutron client
 NEUTRONCLIENT_REPO=${NEUTRONCLIENT_REPO:-${GIT_BASE}/openstack/python-neutronclient.git}
@@ -146,11 +153,11 @@ NEUTRONCLIENT_BRANCH=${NEUTRONCLIENT_BRANCH:-master}

 # consolidated openstack requirements
 REQUIREMENTS_REPO=${REQUIREMENTS_REPO:-${GIT_BASE}/openstack/requirements.git}
-REQUIREMENTS_BRANCH=${REQUIREMENTS_BRANCH:-master}
+REQUIREMENTS_BRANCH=${REQUIREMENTS_BRANCH:-stable/havana}

 # storage service
 SWIFT_REPO=${SWIFT_REPO:-${GIT_BASE}/openstack/swift.git}
-SWIFT_BRANCH=${SWIFT_BRANCH:-master}
+SWIFT_BRANCH=${SWIFT_BRANCH:-stable/havana}
 SWIFT3_REPO=${SWIFT3_REPO:-${GIT_BASE}/fujita/swift3.git}
 SWIFT3_BRANCH=${SWIFT3_BRANCH:-master}

@@ -160,7 +167,7 @@ SWIFTCLIENT_BRANCH=${SWIFTCLIENT_BRANCH:-master}

 # Tempest test suite
 TEMPEST_REPO=${TEMPEST_REPO:-${GIT_BASE}/openstack/tempest.git}
-TEMPEST_BRANCH=${TEMPEST_BRANCH:-master}
+TEMPEST_BRANCH=${TEMPEST_BRANCH:-stable/havana}


 # diskimage-builder
@@ -174,7 +181,7 @@ BM_POSEUR_REPO=${BM_POSEUR_REPO:-${GIT_BASE}/tripleo/bm_poseur.git}
 BM_POSEUR_BRANCH=${BM_POSEUR_BRANCH:-master}

 # a websockets/html5 or flash powered VNC console for vm instances
-NOVNC_REPO=${NOVNC_REPO:-${GIT_BASE}/kanaka/noVNC.git}
+NOVNC_REPO=${NOVNC_REPO:-https://github.com/kanaka/noVNC.git}
 NOVNC_BRANCH=${NOVNC_BRANCH:-master}

 # ryu service
@@ -187,7 +194,7 @@ SPICE_BRANCH=${SPICE_BRANCH:-master}

 # trove service
 TROVE_REPO=${TROVE_REPO:-${GIT_BASE}/openstack/trove.git}
-TROVE_BRANCH=${TROVE_BRANCH:-master}
+TROVE_BRANCH=${TROVE_BRANCH:-stable/havana}

 # trove client library test
 TROVECLIENT_REPO=${TROVECLIENT_REPO:-${GIT_BASE}/openstack/python-troveclient.git}
@@ -122,16 +122,16 @@ fi

 # test empty option
 if ini_has_option test.ini ddd empty; then
-   echo "OK: ddd.empty present"
+    echo "OK: ddd.empty present"
 else
-   echo "ini_has_option failed: ddd.empty not found"
+    echo "ini_has_option failed: ddd.empty not found"
 fi

 # test non-empty option
 if ini_has_option test.ini bbb handlers; then
-   echo "OK: bbb.handlers present"
+    echo "OK: bbb.handlers present"
 else
-   echo "ini_has_option failed: bbb.handlers not found"
+    echo "ini_has_option failed: bbb.handlers not found"
 fi

 # test changing empty option
@@ -55,10 +55,41 @@ def check_indents(line):
            print_error('E003: Indent not multiple of 4', line)


+def starts_multiline(line):
+    m = re.search("[^<]<<\s*(?P<token>\w+)", line)
+    if m:
+        return m.group('token')
+    else:
+        return False
+
+
+def end_of_multiline(line, token):
+    if token:
+        return re.search("^%s\s*$" % token, line) is not None
+    return False
+
+
 def check_files(files):
+    in_multiline = False
+    logical_line = ""
+    token = False
    for line in fileinput.input(files):
-        check_no_trailing_whitespace(line)
-        check_indents(line)
+        # NOTE(sdague): multiline processing of heredocs is interesting
+        if not in_multiline:
+            logical_line = line
+            token = starts_multiline(line)
+            if token:
+                in_multiline = True
+                continue
+        else:
+            logical_line = logical_line + line
+            if not end_of_multiline(line, token):
+                continue
+            else:
+                in_multiline = False
+
+        check_no_trailing_whitespace(logical_line)
+        check_indents(logical_line)


 def get_options():
@@ -22,8 +22,8 @@ run_bm STACKMASTER $HEAD_HOST "ENABLED_SERVICES=g-api,g-reg,key,n-api,n-sch,n-vn
 if [ ! "$TERMINATE" = "1" ]; then
    echo "Waiting for head node ($HEAD_HOST) to start..."
    if ! timeout 60 sh -c "while ! wget -q -O- http://$HEAD_HOST | grep -q username; do sleep 1; done"; then
-      echo "Head node did not start"
-      exit 1
+        echo "Head node did not start"
+        exit 1
    fi
 fi

@@ -229,8 +229,8 @@ EOF

 # (re)start a metadata service
 (
-  pid=`lsof -iTCP@192.168.$GUEST_NETWORK.1:4567 -n | awk '{print $2}' | tail -1`
-  [ -z "$pid" ] || kill -9 $pid
+    pid=`lsof -iTCP@192.168.$GUEST_NETWORK.1:4567 -n | awk '{print $2}' | tail -1`
+    [ -z "$pid" ] || kill -9 $pid
 )
 cd $vm_dir/uec
 python meta.py 192.168.$GUEST_NETWORK.1:4567 &
@@ -268,7 +268,7 @@ if [ "$WAIT_TILL_LAUNCH" = "1" ]; then
    sleep 2

    while [ ! -e "$vm_dir/console.log" ]; do
-      sleep 1
+        sleep 1
    done

    tail -F $vm_dir/console.log &
@@ -105,15 +105,15 @@ if [ -z "$OS_PASSWORD" ]; then
 fi

 if [ -z "$OS_TENANT_NAME" -a -z "$OS_TENANT_ID" ]; then
-   export OS_TENANT_NAME=admin
+    export OS_TENANT_NAME=admin
 fi

 if [ -z "$OS_USERNAME" ]; then
-   export OS_USERNAME=admin
+    export OS_USERNAME=admin
 fi

 if [ -z "$OS_AUTH_URL" ]; then
-   export OS_AUTH_URL=http://localhost:5000/v2.0/
+    export OS_AUTH_URL=http://localhost:5000/v2.0/
 fi

 USER_PASS=${USER_PASS:-$OS_PASSWORD}
@@ -249,7 +249,7 @@ if [ $MODE != "create" ]; then
        for user_id_at_name in `keystone user-list --tenant-id $tenant_id | awk 'BEGIN {IGNORECASE = 1} /true[[:space:]]*\|[^|]*\|$/ {print  $2 "@" $4}'`; do
            read user_id user_name <<< `echo "$user_id_at_name" | sed 's/@/ /'`
            if [ $MODE = one -a "$user_name" != "$USER_NAME" ]; then
-               continue;
+                continue;
            fi
            add_entry "$user_id" "$user_name" "$tenant_id" "$tenant_name" "$USER_PASS"
        done
@@ -24,27 +24,8 @@ source $TOP_DIR/functions

 FILES=$TOP_DIR/files

-# Handle arguments
-
-INSTALL_PIP_VERSION=${INSTALL_PIP_VERSION:-"1.4.1"}
-while [[ -n "$1" ]]; do
-    case $1 in
-        --force)
-            FORCE=1
-            ;;
-        --pip-version)
-            INSTALL_PIP_VERSION="$2"
-            shift
-            ;;
-        --use-get-pip)
-            USE_GET_PIP=1;
-            ;;
-    esac
-    shift
-done
-
-PIP_GET_PIP_URL=https://raw.github.com/pypa/pip/master/contrib/get-pip.py
-PIP_TAR_URL=https://pypi.python.org/packages/source/p/pip/pip-$INSTALL_PIP_VERSION.tar.gz
+PIP_GET_PIP_URL=https://bootstrap.pypa.io/get-pip.py
+LOCAL_PIP="$FILES/$(basename $PIP_GET_PIP_URL)"

 GetDistro
 echo "Distro: $DISTRO"
@@ -60,22 +41,12 @@ function get_versions() {
 }


-function install_get_pip() {
-    if [[ ! -r $FILES/get-pip.py ]]; then
-        (cd $FILES; \
-            curl $PIP_GET_PIP_URL; \
-        )
+function install_get_pip {
+    if [[ ! -r $LOCAL_PIP ]]; then
+        curl -o $LOCAL_PIP $PIP_GET_PIP_URL || \
+            die $LINENO "Download of get-pip.py failed"
    fi
-    sudo python $FILES/get-pip.py
-}
-
-function install_pip_tarball() {
-    (cd $FILES; \
-        curl -O $PIP_TAR_URL; \
-        tar xvfz pip-$INSTALL_PIP_VERSION.tar.gz 1>/dev/null; \
-        cd pip-$INSTALL_PIP_VERSION; \
-        sudo python setup.py install 1>/dev/null; \
-    )
+    sudo -E python $LOCAL_PIP
 }

 # Show starting versions
@@ -86,10 +57,6 @@ get_versions
 # Eradicate any and all system packages
 uninstall_package python-pip

-if [[ -n "$USE_GET_PIP" ]]; then
-    install_get_pip
-else
-    install_pip_tarball
-fi
+install_get_pip

 get_versions
@@ -6,8 +6,8 @@ set -o errexit

 # Make sure only root can run our script
 if [[ $EUID -ne 0 ]]; then
-   echo "This script must be run as root"
-   exit 1
+    echo "This script must be run as root"
+    exit 1
 fi

 # This directory
@@ -31,15 +31,15 @@ apt-get install -y --force-yes $DEPS

 # Install jenkins
 if [ ! -e /var/lib/jenkins ]; then
-   echo "Jenkins installation failed"
-   exit 1
+    echo "Jenkins installation failed"
+    exit 1
 fi

 # Make sure user has configured a jenkins ssh pubkey
 if [ ! -e /var/lib/jenkins/.ssh/id_rsa.pub ]; then
-   echo "Public key for jenkins is missing.  This is used to ssh into your instances."
-   echo "Please run "su -c ssh-keygen jenkins" before proceeding"
-   exit 1
+    echo "Public key for jenkins is missing.  This is used to ssh into your instances."
+    echo "Please run "su -c ssh-keygen jenkins" before proceeding"
+    exit 1
 fi

 # Setup sudo
@@ -96,7 +96,7 @@ PLUGINS=http://hudson-ci.org/downloads/plugins/build-timeout/1.6/build-timeout.h

 # Configure plugins
 for plugin in ${PLUGINS//,/ }; do
-    name=`basename $plugin`   
+    name=`basename $plugin`
    dest=/var/lib/jenkins/plugins/$name
    if [ ! -e $dest ]; then
        curl -L $plugin -o $dest
@@ -0,0 +1,87 @@
+#!/usr/bin/env python
+#
+# Copyright 2014 Hewlett-Packard Development Company, L.P.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+# This is an output filter to filter and timestamp the logs from grenade and
+# devstack. Largely our awk filters got beyond the complexity level which were
+# sustainable, so this provides us much more control in a single place.
+#
+# The overhead of running python should be less than execing `date` a million
+# times during a run.
+
+import argparse
+import datetime
+import re
+import sys
+
+IGNORE_LINES = re.compile('(set \+o|xtrace)')
+HAS_DATE = re.compile('^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3} \|')
+
+
+def get_options():
+    parser = argparse.ArgumentParser(
+        description='Filter output by devstack and friends')
+    parser.add_argument('-o', '--outfile',
+                        help='Output file for content',
+                        default=None)
+    parser.add_argument('-v', '--verbose', action='store_true',
+                        default=False)
+    return parser.parse_args()
+
+
+def skip_line(line):
+    """Should we skip this line."""
+    return IGNORE_LINES.search(line) is not None
+
+
+def main():
+    opts = get_options()
+    outfile = None
+    if opts.outfile:
+        outfile = open(opts.outfile, 'a', 0)
+
+    # otherwise fileinput reprocess args as files
+    sys.argv = []
+    while True:
+        line = sys.stdin.readline()
+        if not line:
+            return 0
+
+        # put skip lines here
+        if skip_line(line):
+            continue
+
+        # this prevents us from nesting date lines, because
+        # we'd like to pull this in directly in grenade and not double
+        # up on devstack lines
+        if HAS_DATE.search(line) is None:
+            now = datetime.datetime.utcnow()
+            line = ("%s | %s" % (
+                now.strftime("%Y-%m-%d %H:%M:%S.%f")[:-3],
+                line))
+
+        if opts.verbose:
+            sys.stdout.write(line)
+            sys.stdout.flush()
+        if outfile:
+            outfile.write(line)
+            outfile.flush()
+
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main())
+    except KeyboardInterrupt:
+        sys.exit(1)
@@ -44,9 +44,9 @@ source $THIS_DIR/xenrc

 xe_min()
 {
-  local cmd="$1"
-  shift
-  xe "$cmd" --minimal "$@"
+    local cmd="$1"
+    shift
+    xe "$cmd" --minimal "$@"
 }

 #
@@ -132,8 +132,8 @@ HOST_IP=$(xenapi_ip_on "$MGT_BRIDGE_OR_NET_NAME")
 # Set up ip forwarding, but skip on xcp-xapi
 if [ -a /etc/sysconfig/network ]; then
    if ! grep -q "FORWARD_IPV4=YES" /etc/sysconfig/network; then
-      # FIXME: This doesn't work on reboot!
-      echo "FORWARD_IPV4=YES" >> /etc/sysconfig/network
+        # FIXME: This doesn't work on reboot!
+        echo "FORWARD_IPV4=YES" >> /etc/sysconfig/network
    fi
 fi
 # Also, enable ip forwarding in rc.local, since the above trick isn't working
@@ -42,69 +42,69 @@ EOF

 get_params()
 {
-  while getopts "hbn:r:l:t:" OPTION;
-  do
-    case $OPTION in
-      h) usage
-         exit 1
-         ;;
-      n)
-         BRIDGE=$OPTARG
-         ;;
-      l)
-         NAME_LABEL=$OPTARG
-         ;;
-      t)
-         TEMPLATE_NAME=$OPTARG
-         ;;
-      ?)
-         usage
-         exit
-         ;;
-    esac
-  done
-  if [[ -z $BRIDGE ]]
-  then
-     BRIDGE=xenbr0
-  fi
+    while getopts "hbn:r:l:t:" OPTION;
+    do
+        case $OPTION in
+            h) usage
+                exit 1
+                ;;
+            n)
+                BRIDGE=$OPTARG
+                ;;
+            l)
+                NAME_LABEL=$OPTARG
+                ;;
+            t)
+                TEMPLATE_NAME=$OPTARG
+                ;;
+            ?)
+                usage
+                exit
+                ;;
+        esac
+    done
+    if [[ -z $BRIDGE ]]
+    then
+        BRIDGE=xenbr0
+    fi

-  if [[ -z $TEMPLATE_NAME ]]; then
-    echo "Please specify a template name" >&2
-    exit 1
-  fi
+    if [[ -z $TEMPLATE_NAME ]]; then
+        echo "Please specify a template name" >&2
+        exit 1
+    fi

-  if [[ -z $NAME_LABEL ]]; then
-    echo "Please specify a name-label for the new VM" >&2
-    exit 1
-  fi
+    if [[ -z $NAME_LABEL ]]; then
+        echo "Please specify a name-label for the new VM" >&2
+        exit 1
+    fi
 }


 xe_min()
 {
-  local cmd="$1"
-  shift
-  xe "$cmd" --minimal "$@"
+    local cmd="$1"
+    shift
+    xe "$cmd" --minimal "$@"
 }


 find_network()
 {
-  result=$(xe_min network-list bridge="$1")
-  if [ "$result" = "" ]
-  then
-    result=$(xe_min network-list name-label="$1")
-  fi
-  echo "$result"
+    result=$(xe_min network-list bridge="$1")
+    if [ "$result" = "" ]
+    then
+        result=$(xe_min network-list name-label="$1")
+    fi
+    echo "$result"
 }


 create_vif()
 {
-  local v="$1"
-  echo "Installing VM interface on [$BRIDGE]"
-  local out_network_uuid=$(find_network "$BRIDGE")
-  xe vif-create vm-uuid="$v" network-uuid="$out_network_uuid" device="0"
+    local v="$1"
+    echo "Installing VM interface on [$BRIDGE]"
+    local out_network_uuid=$(find_network "$BRIDGE")
+    xe vif-create vm-uuid="$v" network-uuid="$out_network_uuid" device="0"
 }


@@ -112,20 +112,20 @@ create_vif()
 # Make the VM auto-start on server boot.
 set_auto_start()
 {
-  local v="$1"
-  xe vm-param-set uuid="$v" other-config:auto_poweron=true
+    local v="$1"
+    xe vm-param-set uuid="$v" other-config:auto_poweron=true
 }


 destroy_vifs()
 {
-  local v="$1"
-  IFS=,
-  for vif in $(xe_min vif-list vm-uuid="$v")
-  do
-    xe vif-destroy uuid="$vif"
-  done
-  unset IFS
+    local v="$1"
+    IFS=,
+    for vif in $(xe_min vif-list vm-uuid="$v")
+    do
+        xe vif-destroy uuid="$vif"
+    done
+    unset IFS
 }


@@ -22,63 +22,63 @@ set -ex
 # By default, don't remove the templates
 REMOVE_TEMPLATES=${REMOVE_TEMPLATES:-"false"}
 if [ "$1" = "--remove-templates" ]; then
-  REMOVE_TEMPLATES=true
+    REMOVE_TEMPLATES=true
 fi

 xe_min()
 {
-  local cmd="$1"
-  shift
-  xe "$cmd" --minimal "$@"
+    local cmd="$1"
+    shift
+    xe "$cmd" --minimal "$@"
 }

 destroy_vdi()
 {
-  local vbd_uuid="$1"
-  local type=$(xe_min vbd-list uuid=$vbd_uuid params=type)
-  local dev=$(xe_min vbd-list uuid=$vbd_uuid params=userdevice)
-  local vdi_uuid=$(xe_min vbd-list uuid=$vbd_uuid params=vdi-uuid)
+    local vbd_uuid="$1"
+    local type=$(xe_min vbd-list uuid=$vbd_uuid params=type)
+    local dev=$(xe_min vbd-list uuid=$vbd_uuid params=userdevice)
+    local vdi_uuid=$(xe_min vbd-list uuid=$vbd_uuid params=vdi-uuid)

-  if [ "$type" == 'Disk' ] && [ "$dev" != 'xvda' ] && [ "$dev" != '0' ]; then
-    xe vdi-destroy uuid=$vdi_uuid
-  fi
+    if [ "$type" == 'Disk' ] && [ "$dev" != 'xvda' ] && [ "$dev" != '0' ]; then
+        xe vdi-destroy uuid=$vdi_uuid
+    fi
 }

 uninstall()
 {
-  local vm_uuid="$1"
-  local power_state=$(xe_min vm-list uuid=$vm_uuid params=power-state)
+    local vm_uuid="$1"
+    local power_state=$(xe_min vm-list uuid=$vm_uuid params=power-state)

-  if [ "$power_state" != "halted" ]; then
-    xe vm-shutdown vm=$vm_uuid force=true
-  fi
+    if [ "$power_state" != "halted" ]; then
+        xe vm-shutdown vm=$vm_uuid force=true
+    fi

-  for v in $(xe_min vbd-list vm-uuid=$vm_uuid | sed -e 's/,/ /g'); do
-    destroy_vdi "$v"
-  done
+    for v in $(xe_min vbd-list vm-uuid=$vm_uuid | sed -e 's/,/ /g'); do
+        destroy_vdi "$v"
+    done

-  xe vm-uninstall vm=$vm_uuid force=true >/dev/null
+    xe vm-uninstall vm=$vm_uuid force=true >/dev/null
 }

 uninstall_template()
 {
-  local vm_uuid="$1"
+    local vm_uuid="$1"

-  for v in $(xe_min vbd-list vm-uuid=$vm_uuid | sed -e 's/,/ /g'); do
-    destroy_vdi "$v"
-  done
+    for v in $(xe_min vbd-list vm-uuid=$vm_uuid | sed -e 's/,/ /g'); do
+        destroy_vdi "$v"
+    done

-  xe template-uninstall template-uuid=$vm_uuid force=true >/dev/null
+    xe template-uninstall template-uuid=$vm_uuid force=true >/dev/null
 }

 # remove the VMs and their disks
 for u in $(xe_min vm-list other-config:os-vpx=true | sed -e 's/,/ /g'); do
-  uninstall "$u"
+    uninstall "$u"
 done

 # remove the templates
 if [ "$REMOVE_TEMPLATES" == "true" ]; then
-  for u in $(xe_min template-list other-config:os-vpx=true | sed -e 's/,/ /g'); do
-    uninstall_template "$u"
-  done
+    for u in $(xe_min template-list other-config:os-vpx=true | sed -e 's/,/ /g'); do
+        uninstall_template "$u"
+    done
 fi
@@ -35,7 +35,12 @@ source $TOP_DIR/lib/apache

 # Get project function libraries
 source $TOP_DIR/lib/baremetal
+source $TOP_DIR/lib/ceilometer
 source $TOP_DIR/lib/cinder
+source $TOP_DIR/lib/keystone
+source $TOP_DIR/lib/glance
+source $TOP_DIR/lib/nova
+source $TOP_DIR/lib/heat
 source $TOP_DIR/lib/horizon
 source $TOP_DIR/lib/swift
 source $TOP_DIR/lib/neutron
@@ -75,21 +80,29 @@ if [[ "$Q_USE_DEBUG_COMMAND" == "True" ]]; then
    teardown_neutron_debug
 fi

-# Shut down devstack's screen to get the bulk of OpenStack services in one shot
-SCREEN=$(which screen)
-if [[ -n "$SCREEN" ]]; then
-    SESSION=$(screen -ls | awk '/[0-9].stack/ { print $1 }')
-    if [[ -n "$SESSION" ]]; then
-        screen -X -S $SESSION quit
-    fi
+# Call service stop
+if is_service_enabled trove; then
+    stop_trove
 fi

-# Shut down Nova hypervisor plugins after Nova
-NOVA_PLUGINS=$TOP_DIR/lib/nova_plugins
-if is_service_enabled nova && [[ -r $NOVA_PLUGINS/hypervisor-$VIRT_DRIVER ]]; then
-    # Load plugin
-    source $NOVA_PLUGINS/hypervisor-$VIRT_DRIVER
-    stop_nova_hypervisor
+if is_service_enabled heat; then
+    stop_heat
+fi
+
+if is_service_enabled ceilometer; then
+    stop_ceilometer
+fi
+
+if is_service_enabled nova; then
+    stop_nova
+fi
+
+if is_service_enabled g-api g-reg; then
+    stop_glance
+fi
+
+if is_service_enabled key; then
+    stop_keystone
 fi

 # Swift runs daemons
@@ -123,6 +136,7 @@ SCSI_PERSIST_DIR=$CINDER_STATE_PATH/volumes/*

 # Get the iSCSI volumes
 if is_service_enabled cinder; then
+    stop_cinder
    cleanup_cinder
 fi

@@ -152,4 +166,13 @@ if is_service_enabled trove; then
    cleanup_trove
 fi

+# Clean up the remainder of the screen processes
+SCREEN=$(which screen)
+if [[ -n "$SCREEN" ]]; then
+    SESSION=$(screen -ls | awk '/[0-9].stack/ { print $1 }')
+    if [[ -n "$SESSION" ]]; then
+        screen -X -S $SESSION quit
+    fi
+fi
+
 cleanup_tmp