OneFlow Services Auto-scaling

A Service Role’s cardinality can be adjusted either manually, or automatically in two ways: based on metrics or based on a schedule.

Overview

When a scaling action starts, the Role and Service enter the SCALING state. In this state, the Role will instantiate or terminate a number of VMs to reach its new cardinality.

A Role with elasticity policies must define a minimum and maximum number of VMs:

"roles": [
    {
      "name": "frontend",
      "cardinality": 1,
      "vm_template": 0,

      "min_vms" : 1,
      "max_vms" : 5,
...

After the scaling, the Role and Service are in the COOLDOWN state for the configured duration. During a scale operation and the cooldown period, other scaling actions for the same or for other Roles are delayed until the Service is RUNNING again.

Set the Cardinality of a Role Manually

The command oneflow scale starts the scalability immediately.

oneflow scale <serviceid> <role_name> <cardinality>

You can force a cardinality outside the defined range with the --force option.

Maintain the Cardinality of a Role

The min_vms attribute is a hard limit, enforced by the elasticity module. If the cardinality drops below this minimum, a scale-up operation will be automatically triggered.

Set the Cardinality of a Role Automatically

Auto-scaling Types

Both elasticity_policies and scheduled_policies elements define an automatic adjustment of the Role cardinality. Three different adjustment types are supported:

• CHANGE: add/subtract the given number of VMs.

• CARDINALITY: set the cardinality to the given number.

• PERCENTAGE_CHANGE: add/subtract the given percentage to the current cardinality.

Attribute	Type	Mandatory	Description
type	string	YES	Type of adjustment. Values: CHANGE, CARDINALITY, PERCENTAGE_CHANGE
adjust	integer	YES	Positive or negative adjustment. Its meaning depends on type.
min_adjust_step	integer	NO	Optional parameter for PERCENTAGE_CHANGE adjustment type. If present, the policy will change the cardinality by at least the number of VMs set in this attribute.

Auto-scaling Based on Metrics

Each Role can have an array of elasticity_policies. These policies define an expression that will trigger a cardinality adjustment.

These expressions can use performance data from

• The VM guest. Using the OneGate server, applications can send custom monitoring metrics to OpenNebula.

• The VM, at hypervisor level. The Virtualization Drivers return information about the VM, such as CPU, NETTX and NETRX.

"elasticity_policies" : [
  {
    "expression" : "ATT > 50",
    "type" : "CHANGE",
    "adjust" : 2,

    "period_number" : 3,
    "period" : 10
  },
  ...
]

The expression can use VM attribute names, float numbers, and logical operators (!, &, |). When an attribute is found, it will take the average value for all the running VMs that contain that attribute in the Role. If none of the VMs contain the attribute, the expression will evaluate to false.

The attribute will be looked for in /VM/USER_TEMPLATE, /VM/MONITORING, /VM/TEMPLATE and /VM, in that order. Logical operators have the usual precedence.

Attribute	Type	Mandatory	Description
expression	string	YES	Expression to trigger the elasticity.
period_number	integer	NO	Number of periods that the expression must be true before the elasticity is triggered.
period	integer	NO	Duration, in seconds, of each period in period_number.

Auto-scaling Based on a Schedule

Combined with the elasticity policies, each Role can have an array of scheduled_policies. These policies define a time or a time recurrence, and a cardinality adjustment.

"scheduled_policies" : [
  {
    // Set cardinality to 2 each 10 minutes
    "recurrence" : "*/10 * * * *",

    "type" : "CARDINALITY",
    "adjust" : 2
  },
  {
    // +10 percent at the given date and time
    "start_time" : "2nd oct 2017 15:45",

    "type" : "PERCENTAGE_CHANGE",
    "adjust" : 10
  }
]

Attribute	Type	Mandatory	Description
recurrence	string	NO	Time for recurring adjustments. Time is specified with the Unix cron syntax.
start_time	string	NO	Exact time for the adjustment.

Visualize in the CLI

The oneflow show command show the defined policies:

SERVICE 7 INFORMATION
...

ROLE frontend
ROLE STATE          : SCALING
CARNIDALITY         : 4
VM TEMPLATE         : 0
NODES INFORMATION
 VM_ID NAME                    STAT UCPU    UMEM HOST                       TIME
     4 frontend_0_(service_7)  runn    0   74.2M host03                 0d 00h04
     5 frontend_1_(service_7)  runn    0  112.6M host02                 0d 00h04
     6                         init           0K                        0d 00h00
     7                         init           0K                        0d 00h00

ELASTICITY RULES
MIN VMS             : 1
MAX VMS             : 5

ADJUST       EXPRESSION                                        EVALUATION PERIOD
+ 2          (ATT > 50) && !(OTHER_ATT = 5.5 || ABC <= 30)     0 / 3         10s
- 10 % (2)   ATT < 20                                          0 / 1          0s

ADJUST       TIME
= 6          0 9 * * mon,tue,wed,thu,fri
= 10         0 13 * * mon,tue,wed,thu,fri
= 2          30 22 * * mon,tue,wed,thu,fri


LOG MESSAGES
06/10/13 18:22 [I] New state: DEPLOYING
06/10/13 18:22 [I] New state: RUNNING
06/10/13 18:26 [I] Role frontend scaling up from 2 to 4 nodes
06/10/13 18:26 [I] New state: SCALING

Interaction with Individual VM Management

All the VMs created by a Service can be managed as regular VMs. When VMs are monitored in an unexpected state, this is what OneFlow interprets:

• VMs in the final done state are cleaned from the Role. They do not appear in the nodes information table, and the cardinality is updated to reflect the new number of VMs. This can be seen as an manual scale-down action.

• VMs in poweroff, unknown or failed are in an anomalous state, and the user must be notified. The Role and Service are set to the WARNING state.

Examples

/*
Testing:

1) Update one VM template to contain
ATT = 40
and the other VM with
ATT = 60

Average will be 50, true evaluation periods will not increase in CLI output

2) Increase first VM ATT value to 45. True evaluations will increase each
10 seconds, the third time a new VM will be deployed.

3) True evaluations are reset. Since the new VM does not have ATT in its
template, the average will be still bigger than 50, and new VMs will be
deployed each 30s until the max of 5 is reached.

4) Update VM templates to trigger the scale down expression. The number of
VMs is adjusted -10 percent. Because 5 * 0.10 < 1, the adjustment is rounded to 1;
but the min_adjust_step is set to 2, so the final adjustment is -2 VMs.
*/
{
  "name": "Scalability1",
  "deployment": "none",
  "roles": [
    {
      "name": "frontend",
      "cardinality": 2,
      "vm_template": 0,

      "min_vms" : 1,
      "max_vms" : 5,

      "elasticity_policies" : [
        {
          // +2 VMs when the exp. is true for 3 times in a row,
          // separated by 10 seconds
          "expression" : "ATT > 50",

          "type" : "CHANGE",
          "adjust" : 2,

          "period_number" : 3,
          "period" : 10
        },
        {
          // -10 percent VMs when the exp. is true.
          // If 10 percent is less than 2, -2 VMs.
          "expression" : "ATT < 20",

          "type" : "PERCENTAGE_CHANGE",
          "adjust" : -10,
          "min_adjust_step" : 2
        }
      ]
    }
  ]
}

{
  "name": "Time_windows",
  "deployment": "none",
  "roles": [
    {
      "name": "frontend",
      "cardinality": 1,
      "vm_template": 0,

      "min_vms" : 1,
      "max_vms" : 15,

      // These policies set the cardinality to:
      //  6 from  9:00 to 13:00
      // 10 from 13:00 to 22:30
      //  2 from 22:30 to 09:00, and the weekend

      "scheduled_policies" : [
        {
          "type" : "CARDINALITY",
          "recurrence" : "0 9 * * mon,tue,wed,thu,fri",
          "adjust" : 6
        },
        {
          "type" : "CARDINALITY",
          "recurrence" : "0 13 * * mon,tue,wed,thu,fri",
          "adjust" : 10
        },
        {
          "type" : "CARDINALITY",
          "recurrence" : "30 22 * * mon,tue,wed,thu,fri",
          "adjust" : 2
        }
      ]
    }
  ]
}