Sunstone for Large Deployments

Low to medium size enterprise clouds will typically deploy Sunstone on a single machine with the other OpenNebula daemons as part. However, this simple deployment can be extended by

  • Isolating access from web clients to the Sunstone server. This can be achieved by deploying the Sunstone server on a separate machine.

  • Improving scalability of the server for large user pools, usually by deploying Sunstone as a separate application on one or more hosts.

This guide introduces various deployment options to achieve this. Check also the API Scalability guide for tips on how to improve Sunstone and OpenNebula Daemon performance.

Deploy in Webserver

Self-contained deployment of Sunstone (using opennebula-sunstone system service) is sufficient for small to medium installations. This is no longer enough when the service has lots of concurrent users and the number of objects in the system is high (for example, more than 2000 simultaneous Virtual Machines).

The Sunstone server is implemented as a Rack server. This makes it suitable to run in any web server that supports this protocol. In the Ruby world this is the standard supported by most web servers. We can now select web servers that support spawning multiple processes, like Unicorn, or embedding the service inside Apache HTTP Server or Nginx using the Passenger module. Another benefit is the ability to run Sunstone on several servers and balance the load between them.

Warning

Deploying Sunstone behind a proxy in a federated environment requires specific configuration to properly handle the Sunstone headers required by the Federation.

  • nginx: enable underscores_in_headers on; and proxy_pass_request_headers on;

Disable opennebula-sunstone system service

The opennebula-sunstone system service must be stopped and disabled when Sunstone is deployed in a web server.

systemctl stop opennebula-sunstone
systemctl disable opennebula-sunstone

Memcached

Sunstone needs to store user sessions so it doesn’t ask for user/password for every action. By default Sunstone is configured to use memory sessions meaning that the sessions are stored in the process memory. Thin and webrick web servers do not spawn new processes but new threads, and all of them have access to that session pool. When using more than one process for the Sunstone server, there must be a service that stores the session information and can be accessed by all the processes. OpenNebula is using the memcached server for this purpose. It comes with most distributions and its default configuration should be OK.

Change the value of :sessions to memcache in the Sunstone configuration (/etc/one/sunstone-server.conf).

noVNC

If you want to use noVNC you need to have it running. You can start this service with the command:

systemctl enable opennebula-novnc
systemctl start  opennebula-novnc

Filesystem Permissions

Another thing you have to take into account is the user under which the server will run. The installation sets the permissions for the oneadmin user and group, and files like the Sunstone configuration and credentials cannot be read by other users.

chmod a+x /var
chmod a+x /var/lib
chmod a+x /var/lib/one
chmod a+x /var/lib/one/.one
chmod a+x /var/lib/one/sunstone

Marketplace

If you are planning to support downloading of Appliances from Marketplace, the Sunstone server(s) will need access to the Marketplace Back-ends. Also, apply following recommendations if using Phusion Passenger :

If you are using a Back-end other than Passenger, please adapt these recommendations accordingly.

We advise using Apache/Passenger in your installation, but we’ll also present the other options. For more information on Rack and web servers that support it, you can check the Rack documentation page. Alternatively, you can check a list of Ruby web servers.

Deploy with Unicorn (Legacy)

The Unicorn is a multi-process Ruby webserver. The installation is achieved using RubyGems tools (or with your package manager if it is available). E.g.:

gem install unicorn

In the directory where Sunstone files reside (/usr/lib/one/sunstone), there is a file config.ru specifically for Rack applications that instructs how to run the application. To start a new server using unicorn, you can run this command from that directory:

$ unicorn -p 9869

The default Unicorn configuration should be fine for most installations, but a configuration file can be created to tune it. For example, to tell Unicorn to spawn four processes and redirect standard error output to /tmp/unicorn.log, we can create a file called unicorn.conf with the following content:

worker_processes 4
logger debug
stderr_path '/tmp/unicorn.log'

and start and daemonize the Unicorn server this way:

$ unicorn -d -p 9869 -c unicorn.conf

Note

See the complete Unicorn configuration options.

Deploy in Dedicated Host

By default, the Sunstone server is configured to run on the Single Front-end alongside the other OpenNebula components. You can also install and run the Sunstone server on a different dedicated machine.

  • Install only the Sunstone server package on the machine that will be running the server.

  • Ensure the :one_xmlprc: option in /etc/one/sunstone-server.conf points to the endpoint where OpenNebula Daemon is running (e.g., http://opennebula-oned:2633/RPC2). You can also leave it undefined and export the ONE_XMLRPC environment variable.

  • (Optional) On host running OpenNebula Daemon, enable ZeroMQ to listen to non-localhost address. In /etc/one/oned.conf in HM_MAD/ARGUMENTS replace -b 127.0.0.1 with your IP address accessible by Sunstone from a different machine (e.g., -b 192.168.0.1). Update the endpoints accordingly in /etc/one/onehem-server.conf in parameters :subscriber_endpoint and :replier_endpoint. IMPORTANT: This endpoint is not secure and should be available only through private IPs (unreachable from outside). Set the IP carefully, never set wildcard address 0.0.0.0! Sensitive information from the OpenNebula might leak!!!

  • (Optional) In Sunstone configuration set :subscriber_endpoint for the connections to OpenNebula ZeroMQ endpoint above.

  • (Optional) In Sunstone configuration set FireEdge endpoints :public_fireedge_endpoint and :private_fireedge_endpoint.

  • Provide the serveradmin and oneadmin credentials in the /var/lib/one/.one/.

  • If you want to upload files to OpenNebula, you will have to share the uploads directory (/var/tmp by default) between Sunstone and oned. Some servers do not take into account the TMPDIR environment variable, in which case this directory must be defined in the configuration file (:tmpdir). It may also be necessary to set it in Passenger (client_body_temp_path).

  • For OneFlow service to work you will need to set :oneflow_server:. The value will be pointing to the current OneFlow server, e.g.: http://opennebula-oned:2474

  • (Optional) Share /var/log/one across Sunstone and OpenNebula Daemon machines to have access to Virtual Machine logs.

Consider also combination with deployment in webserver above.

Multiple Hosts

You can run Sunstone on several servers and use a load balancer that connects to them. Make sure you are using memcache for sessions and both Sunstone servers connect to the same memcached server. To do this, change the parameter :memcache_host in the configuration file. Also make sure that both Sunstone instances connect to the same OpenNebula server.

Configuring an SSL Proxy

OpenNebula Sunstone runs natively just on normal HTTP connections. If the extra security provided by SSL is needed, a proxy can be set up to handle the SSL connection that forwards the request to the Sunstone server and returns the answer to the client.

This set up needs:

  • A server certificate for the SSL connections

  • An HTTP proxy that understands SSL

  • OpenNebula Sunstone configuration to accept requests from the proxy

If you want to try out the SSL setup easily, the following lines provide an example to set a self-signed certificate to be used by a web server configured to act as an HTTP proxy to a correctly configured OpenNebula Sunstone.

Let’s assume the server where the proxy is going to be started is called cloudserver.org. Therefore, the steps are:

Step 1: Server Certificate (Snakeoil)

We are going to generate a snakeoil certificate. If using an Ubuntu system follow the next steps (otherwise your mileage may vary, but not a lot):

  • Install the ssl-cert package

apt-get install ssl-cert

  • Generate the certificate

/usr/sbin/make-ssl-cert generate-default-snakeoil

  • As we are using lighttpd, we need to append the private key to the certificate to obtain a server certificate valid to lighttpd

cat /etc/ssl/private/ssl-cert-snakeoil.key /etc/ssl/certs/ssl-cert-snakeoil.pem > /etc/lighttpd/server.pem

Step 2: SSL HTTP Proxy

lighttpd

You will need to edit the /etc/lighttpd/lighttpd.conf configuration file and

  • Add the following modules (if not present already)

mod_access
mod_alias
mod_proxy
mod_accesslog
mod_compress

  • Change the server port to 443 if you are going to run lighttpd as root, or any number above 1024 otherwise:

server.port               = 8443

  • Add the proxy module section:

#### proxy module
## read proxy.txt for more info
proxy.server               = ( "" =>
                                ("" =>
                                 (
                                   "host" => "127.0.0.1",
                                   "port" => 9869
                                 )
                                 )
                             )


#### SSL engine
ssl.engine                 = "enable"
ssl.pemfile                = "/etc/lighttpd/server.pem"

The host must be the server hostname of the computer running the Sunstone server, and the port the one that the Sunstone Server is running on.

nginx

You will need to configure a new virtual host in nginx. Depending on the operating system and the method of installation, nginx loads virtual host configurations from either /etc/nginx/conf.d or /etc/nginx/sites-enabled.

  • A sample cloudserver.org virtual host is presented next:

# No squealing.
server_tokens off;

#### OpenNebula Sunstone upstream
upstream sunstone  {
        server 127.0.0.1:9869;
}

#### OpenNebula FireEdge upstream
upstream fire-edge {
        server 127.0.0.1:2616;
}

#### cloudserver.org HTTP virtual host
server {
        listen 80;
        server_name cloudserver.org;

        ### Permanent redirect to HTTPS (optional)
        return 301 https://$server_name:8443;
}

#### cloudserver.org HTTPS virtual host
server {
        listen 8443;
        server_name cloudserver.org;

        ### SSL Parameters
        ssl on;
        ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
        ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

        ### Proxy requests to upstream
        location / {
                proxy_pass              http://sunstone;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header        X-Forwarded-Proto $scheme;
        }

        location /fireedge {
                proxy_pass http://fire-edge/fireedge;
                proxy_redirect off;
                log_not_found off;
                proxy_buffering off;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $http_host;
                proxy_set_header X-Forwarded-FOR $proxy_add_x_forwarded_for;
                access_log off;
        }
}

The IP address and port number used in upstream must be the ones the server Sunstone is running on. On typical installations the nginx master process is run as user root so you don’t need to modify the HTTPS port.

haproxy

To access sunstone via HTTPS via haproxy, please check the frontend and backend configurations

  • About the frontend configuration, please remember that the SSL certificate and its private key must be concatenated. Supposing that the certificate and key for your server are host.crt and host.key, and you want to set the resulting file in /etc/certs/host.pem the commands would be these (the right permissions should be set too)

# cat host.crt host.key | tee /etc/certs/host.pem
# chmod 600 /etc/certs/host.pem

We will suppose that the CA certificate is in the file /etc/certs/ca.crt. The CA doesn’t need to be concatenated with its private key.

Once that is set, a frontend valid configuration would look like this, sending the requests to the server we configure as BACKEND_ONE

frontend FRONTEND_ONE
    mode http
    bind *:80
    bind *:443 ssl crt /etc/certs/cert.pem ca-file /etc/certs/ca.crt verify optional crt-ignore-err all
    # To redirect the requests from HTTP to HTTPS uncomment the following line
    # http-request redirect scheme https unless { ssl_fc }
    default_backend BACKEND_ONE

  • The backend configuration is different depending if you want to use core or x509 authentication.

    • If you want to use core authentication (based on username and password), the frontend would look like this (names in capital letters may be needed to be changed)

    # haproxy backend config for Sunstone using core auth driver
backend BACKEND_ONE
   mode http
   option forwardfor
   server ONE_CORE 127.0.0.1:9869

    • In case you want to use x509 certificates for the authentication in sunstone, you need to pass some extra headers in order to make it work correctly

    # haproxy backend config for Sunstone using x.509 auth driver
backend BACKEND_ONE
   mode http
    option forwardfor
    http-request set-header SSL_CLIENT_CERT       -----BEGIN\ CERTIFICATE-----\n%{+Q}[ssl_c_der,base64]\n-----END\ CERTIFICATE-----
    http-request set-header SSL_INFO              %[ssl_c_s_dn(cn)]:%[ssl_c_s_dn]
    http-request set-header AUTHORIZATION         Basic\ %[req.fhdr(SSL_INFO),base64]
    http-request del-header SSL_INFO
    server ONE_X509 127.0.0.1:9869

Step 3: Sunstone Configuration

Edit /etc/one/sunstone-server.conf to listen at localhost:9869.

:host: 127.0.0.1
:port: 9869

Once the proxy server is started, OpenNebula Sunstone requests using HTTPS URIs can be directed to https://cloudserver.org:8443, that will then be unencrypted, passed to localhost, port 9869, satisfied (hopefully), encrypted again and then passed back to the client.

Also, configure TLS settings for noVNC in sunstone-server.conf in the following way:

:vnc_proxy_port: 29876
:vnc_proxy_support_wss: only
:vnc_proxy_cert: /etc/one/ssl/opennebula-certchain.pem
:vnc_proxy_key: /etc/ssl/certs/ssl-cert-snakeoil.pem
:vnc_proxy_ipv6: false

Note

If using a self-signed certificate, the connection to VNC windows in Sunstone might fail. Either get a real certificate or manually accept the self-signed one in your browser before trying it with Sunstone. Now, VNC sessions should show “encrypted” in the title. You will need to have your browser trust that certificate for both the 443 and 29876 ports on the OpenNebula IP or FQDN.