..
      Copyright 2015 Mirantis Inc. All Rights Reserved.

      Licensed under the Apache License, Version 2.0 (the "License"); you may
      not use this file except in compliance with the License. You may obtain
      a copy of the License at

          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
      License for the specific language governing permissions and limitations
      under the License.

.. _tutorial_step_4_adding_success_criteria_for_benchmarks:

Step 4. Adding success criteria (SLA) for subtasks
==================================================

.. contents::
   :local:

SLA - Service-Level Agreement (Success Criteria)
------------------------------------------------

Rally allows you to set success criteria (also called *SLA - Service-Level
Agreement*) for every subtask. Rally will automatically check them for you.

To configure the SLA, add the *"sla"* section to the configuration of the
corresponding subtask (the check name is a key associated with its target
value). You can combine different success criteria:

.. code-block:: json

    {
        "version": 2,
        "title": "Boot and delete servers with success criteria",
        "subtasks": [
            {
                "title": "Boot and delete a single server",
                "scenario": {
                    "NovaServers.boot_and_delete_server": {
                        ...
                    }
                },
                "runner": {
                    ...
                },
                "contexts": {
                    ...
                },
                "sla": {
                    "max_seconds_per_iteration": 10,
                    "failure_rate": {
                        "max": 25
                    }
                }
            }
        ]
    }

Such configuration will mark the **NovaServers.boot_and_delete_server**
task scenario as not successful if either some iteration took more than 10
seconds or more than 25% iterations failed.


Checking SLA
------------
Let us show you how Rally SLA work using a simple example based on
**Dummy scenarios**. These scenarios actually do not perform any
OpenStack-related stuff but are very useful for testing the behaviors of Rally.
Let us put in a new task, *test-sla.json*, 2 scenarios -- one that does nothing
and another that just throws an exception:

.. code-block:: json

    {
        "version": 2,
        "title": "Checking SLA with dummy scenarios",
        "subtasks": [
            {
                "title": "Dummy scenario that does nothing",
                "scenario": {
                    "Dummy.dummy": {}
                },
                "runner": {
                    "constant": {
                        "times": 5,
                        "concurrency": 2
                    }
                },
                "contexts": {
                    "users": {
                        "tenants": 3,
                        "users_per_tenant": 2
                    }
                },
                "sla": {
                    "failure_rate": {"max": 0.0}
                }
            },
            {
                "title": "Dummy scenario that throws an exception",
                "scenario": {
                    "Dummy.dummy_exception": {}
                },
                "runner": {
                    "constant": {
                        "times": 5,
                        "concurrency": 2
                    }
                },
                "contexts": {
                    "users": {
                        "tenants": 3,
                        "users_per_tenant": 2
                    }
                },
                "sla": {
                    "failure_rate": {"max": 0.0}
                }
            }
        ]
    }

Note that both scenarios in these tasks have the **maximum failure rate of 0%**
as their **success criterion**. We expect that the first scenario is going
to pass this criterion while the second will fail it. Let's start the task:


.. code-block:: bash

   rally task start test-sla.json

After the task completes, run *rally task sla_check* to check the results again
the success criteria you defined in the task:

.. code-block:: console

   $ rally task sla_check
   +-----------------------+-----+--------------+--------+-------------------------------------------------------------------------------------------------------+
   | subtask               | pos | criterion    | status | detail                                                                                                |
   +-----------------------+-----+--------------+--------+-------------------------------------------------------------------------------------------------------+
   | Dummy.dummy           | 0   | failure_rate | PASS   | Maximum failure rate percent 0.0% failures, minimum failure rate percent 0% failures, actually 0.0%   |
   | Dummy.dummy_exception | 0   | failure_rate | FAIL   | Maximum failure rate percent 0.0% failures, minimum failure rate percent 0% failures, actually 100.0% |
   +-----------------------+-----+--------------+--------+-------------------------------------------------------------------------------------------------------+

Exactly as expected.


SLA in task report
------------------

SLA checks are nicely visualized in task reports. Generate one:

.. code-block:: bash

   rally task report --out=report_sla.html --open

SubTask that have passed SLA have a green check on the overview
page:

.. image:: ../../images/Report-SLA-Overview.png
   :align: center

Somewhat more detailed information about SLA is displayed on the subtask
pages:

.. image:: ../../images/Report-SLA-Scenario.png
   :align: center

Success criteria present a very useful concept that enables not only to analyze
the outcome of your tasks, but also to control their execution. In
:ref:`one of the next sections <tutorial_step_6_aborting_load_generation_on_sla_failure>`
of our tutorial, we will show how to use SLA to abort the load generation
before your OpenStack goes wrong.
