Brand Claim Brand Claim

Imagefactory Introduction

Overview

Public Linux images on Open Telekom Cloud (OTC) come from several sources:

  • This OTC Image Factory: Images built from repositories using kiwi or disk-image-builder
  • Community Images with no or only minor changes
  • Images from Huawei that have uvp-tools and are used in a similar form in HEC (Huawei Enterprise Cloud)
  • Vendor images that have been built by the OS vendor according to our guidelines for OTC

Of course, customers can also build and register private images.

Image Factory Images

Overview

The images carry the prefix Standard_ for non-commercial Linux versions and Enterprise_ for Linux versions that come with a subscription from a Linux OS vendor. We currently have images for openSUSE, CentOS, SLES and Oracle Linux in this category (RedHat to come soon).

The Standard images use a root disk size of 4GiB by default and are registered with 4GiB minimal root disk size. (They used to be 2GiB only but still were registered with 4GiB prior to Dec 2016.) You can work with these or configure a larger root disk (e.g. 8GiB) at VM creation in case you want to inject significant amount of packages. The Enterprise images are 4GB for Oracle Linux and SLES (you can again chose much larger disks if you want) and 20GiB for the SLES11 extended (previously called SAP) images due to using LVM. The (growpart and resizefs cloud-init modules), while on the LVM images, a larger disk will leave space to enlarge the VG and LVs manually.

Configuration

All images are automatically built nightly from mirrored upstream repositories using kiwi7 (openSUSE, SLES, CentOS, RedHat, Oracle, EulerOS) or diskimagebuilder (Debian, Fedora) and are then subjected to an automated testsuite. They have been subjected to standard configuration and hardening measures. The images contain the needed xen drivers (xen-classic or pvops), kvm virtio drivers, uvp-monitor (see) below and most have the cloud management tools (such as otc-tools) that work with OTC preinstalled.

They are all cloud-init enabled and have a standard user linux with a RANDOM password (via chpasswd cloud-init module) configured for console (remote noVNC console) access that has full passwordless sudo power. This password is displayed as a login hint at the noVNC console (it is the emergency login method). (Before 2016-12, the passwords were set to a uniform cloud.1234.)

sshd is configured for remote login, but no password authentication nor root login is permitted, which the testsuite validates. (Remember that you need to open up port 22 in the security group if you want to login from machines outside the same SG.) sshd key injection is configured for the user linux. Please ensure that a strong password is set for the user linux if you really want to enable ssh password authentication. (It is a bad idea for VMs exposed to the internet: You will see several attempts per minute to guess your password from attackers in the internet as of early 2016.)

The images have update repositories registered to make it easy for you to keep up with security updates from the OS vendors. We have repository mirrors in the OTC which will be configured unless you boot VMs in BYOL (bring your own license) mode. It is absolutely recommended to automatically deploy these to keep your cloud application secure. Despite this, we have not at this time configured automatic updates in the images. The reason is that many cloud apps instead chose to have relatively short-lived VMs and redeploy using new VMs using the latest images (which already include all the updates) regularly instead. (We support this model using the _latest images, see below.)

The image configuration files and the .qcow2 files are signed with the ImageFactory GPG key 2048R/03067050 or (starting on 2018-06-01) 2048R/C4C85D49.

OTC Image Builder key:

pub   2048R/03067050 2016-01-07 [expires: 2018-01-06]
      Key fingerprint = 224C DD82 F3E3 8F60 D690  D891 0CDB 8F9F 0306 7050
uid                  OTC Image Builder <imgfactory@otc.t-systems.com>
sub   2048R/D8876977 2016-01-07 [expires: 2018-01-06]
pub   2048R/C4C85D49 2018-01-06 [expires: 2020-01-06]
      Key fingerprint = 85C8 D136 9BA0 A102 47E5  F8A3 B52D CBA9 C4C8 5D49
uid       [ultimate] OTC ImageBuilder (OTC ImageFactory image signing key 2018/19) <imagefactory@otc.t-systems.com>
sub   2048R/F7DA39F9 2018-01-06 [expires: 2020-01-06]

cloud-init

cloud-init runs on the bootup of a VM to configure and customize the VM. It does things like resizing the partition and root filesystem to fill the complete system disk, seeding the VMs PRNG, setting up users and passwords, generating the ssh host key (if not already set or if the instance identity changed), injecting ssh keys (via the cloud metadata server) and optionally doing a lot more, such as installing packages or package updates or subjecting your VM to a configuration management system.

Since 2016-07-25, OTC supports injecting user-data to the metadata server for cloud-init to pick up and customize the VM on the first bootup. So the workaround with injecting files to /etc/cloud/cloud.cfg.d/20_custom_data.cfg is no longer needed.

uvp-monitor and other drivers

OTC has some nice extra functionality, such as collecting monitoring data (CPU, disk and network load) from the VMs, memory and CPU hot-add, disk and VM snapshots, live-migration enhancements. For these features to work, a tool called uvp-monitor is running inside the VMs collecting the data and feeding it to the host (via xenstore/xenbus). The uvp-monitor package depends on xenbus availability in the guest, which normally works out of the box with the standard distribution's xen PV or PVOPS drivers. For CentOS-6 and OracleLinux-6, extra kernel drivers (kmod-uvpmod package) are required, for openSUSE42.1, the kernel drivers have been patched to offer a device file to avoid a deadlock. The uvp-monitor package is enabled on all ImageFactory images except RedHat Enterrpise Linux.

uvp-monitor as well as the kernel driver is available under the GNU GPL and can be found in the home:garloff:OTC project on openSUSE build service.

To provide good performance for the High-Performance Instances h1.*, the intel ixgbevf driver version 2.16.4 is included in most of the images; RedHat again does not allow a custom package and thus has slightly lower performance using the shipped 2.12.1k driver version. On the openSUSE, SLES, CentOS, Oracle Linux, EulerOS images, the driver is included as a cleanly package KMP/kmod, so it has the dependencies to kernel symbols reflected in the RPM, survives normal kernel security updates and can be updated itself as needed.

The Mellanox drivers for the Infiniband stack for the h2. and hl1. flavors are not included in the images; the vendordata mechanism is used to register the mirrored Mellanox repositories and install the needed packages on the first boot of a VM on these flavors.

Using the images

You can download the .qcow2 files from the ImageFactory, upload them to your Object Storage (using S3Browser, OBSbrowser or simply s3 from libs3) and register them as private image for your tenant in the Image Management Service (Web Interface). Assign at least 4GiB image size for the images (at least 20GiB for the SLES extended images and 96GiB for the SLES HANA images as those have an LVM setup assuming 20GiB/96GiB respectively.) You can of course also register the images via the glance API.

The images are also registered as public images using glance for general usage. We regularly provide images with the date _latest -- you can search for the IDs of those using the glance image-list command and then use those to deploy VMs in your automation scripts using the OpenStack/OTC API. We also provide monthly versions with a date attached and will keep these around for at least two years. (Note that we may decide to pusblish extra versions if important security issues pop up that make us trigger additional image publication.)

Update policy for public images from the ImageFactory

We used to register images from the image factory twice each, once with a _date stamp and a _latest image.

We have changed the policy on 2018-04-01, now there only exist _latest, _prev, and hidden (but referencable by UUID) older images (with a datestamp in the name).

The _latest images get renewed roughly once a month -- however we sometimes push out new _latest images in between to address a bug or enable a new feature.

Image Table

The following Table gives an overview about the images and the supported flavors.

Known issues

  • On CentOS-6.x, Oracle-6.x, and SLES11, the partition growth from cloud-init is only accepted by the kernel after a reboot, which means that the initial VM deployment takes an extra minute due to an extra reboot.
  • Password injection is now done via cloud-init user-data. See this announcement to see how this can be achieved.
  • The LVM setup for the root disk on the SLES extended and HANA images might suggest it is a good idea to enhance the storage with further disks and extend the volume group to enlarge the disks. This can be done, but for resilient cloud-ready applications, it is recommended to design stateless VMs, where the root disk does not carry any data that is not automatically injected via cloud-init or config management systems (such as ansible, chef, puppet, saltstack) from the outside.
  • While the images (except SLES SAP) easily fit within a 2GB root filesystem, they are built and registered with a min-disk size of 4GB currently. This was due to a limitation in the platform that allows larger root disks only from a base size of 4GB onwards. The limitation is gone meanwhile, but we do consider 4GB a good compromise of still being lightweight and leaving enough space for customers to install software. Really small services tend to use containers these days ... but let us know if you have a need for really small images.
  • As of Dec 2016, the build Oracle Linux 6.x image does not work anymore; we're still investigating why this is the case. Just be aware that Oracle Linux 6 is thus currently outdated and assess the security risks.
  • The support for baremetal images is achieved via a special package bms-network-config. It compensates for a shortcoming of cloud-init to parse the bonding (LACP) network setup from the ConfigDrive DataSource. bms-network-setup does the network config and then hands over to cloud-init for the other cloud-init configuration tasks. However, there is some overlap that can lead to network setup failures on bare metal (physical.*) instances which is currently under investigation.
  • The Mellanox Infiniband drivers are registered and injected on booting h2. and hl1. VMs. This delays the boot process between 20s (SLES) and two minutes (CentOS). The reason for the long delay on CentOS is that yum is relatively slow and that it currently needs an extra reboot. The latter is under investigation. On SLES, there is a limiation in the DHCP client that results in the ipoib InfiniBand network adapter ib0 not receiving an IP address, so the IP configuration needs to revert to static IP adresses for these adapters. There is a discussion with SUSE and Huawei to improve this in hte future.

SLES11_SP4_extended, SLES11_SP4_SAPHANA and SLES12_SP1_SAPHANA

These three images are derived from the standard SLES, but have a significantly different configuration. They all use LVM with a multi-filesysten setup. LVM enables the flexibility to resize filesystems.

The SLES11_SP4_extended images contains a lot of extra software -- the former name SLES11_SP4_SAP hints at what this was meant for: The software recommended for SAP deployments.

The SLES11_SP4_SAPHANA and SLES12_SP1_SAPHANA images are specialized image for the special SAPHANA flavor available in OTC. They are built using the SLES for SAP repositories from SUSE and contain a setup suitable for SAP HANA deployment, using two high-performance local disks (using a paravirtualized SCSI driver) in addition to the normal shared root filesystem as well as high-performance (SRIOV) networking. You should use the SAPHANA (e1, e2) flavors for these images.

Images and flavors

Initially there was only the "normal" flavors: c1,c2,s1,m1 (aka compute1/2, general purpose, high mem) which had no special requirements on the images beyond the xen drivers (and the optional uvp-monitor). These days, there are flavors offering 10Gbps SRIOV networking via the ixgbevf driver: d1,e1,e2,h1 (aka diskintensive, SAPHANA 1/2, highperformance), local disks: d1,e1,e2 or vGPU support: g1 (vGPU M60).

Most of the Linux images have the appropriate SRIOV network driver to work with the d1,h1 flavors, though qualification is still ongoing, so the currently selectable images may still be a bit more limited than it will be in the future.

The image capabilities are recorded in a bitmask (__image_capabilities)) that is then translated into flavor support flags (__is_support_XXX) which is used by the platform to filter images for the flavors. (The platform in turn uses host aggregates to ensure the hosts have the right capabilities for the flavors, but this is not normally visible to the end user.)

More flavors will emerge in the future, though we try to avoid too many types which would potentially be too confusing.

Only the "normal" (c1,c2,s1,m1) flavors and the new standard KVM (m2,s2) flavors support live migration; this means that host maintenance that requires host shutoff or reboots on hosts carrying the other flavors will result in VMs being stopped and restarted (cold migration). The reason is that flavors with hardware pass-through are very difficult or impossible to live-migrate as there is no sufficient standardized hardware state-transfer mechanism available. Customers need to build cloud-ready/cloud-native application architectures that can deal with this if they want to achieve highest levels of availability. (Obviously, we also recommend cloud-ready/native application architecture for the "normal" flavors, as a scale-out cloud platform does by definition not provide HA for VMs and OTC is no different, even if live-migrations help a bit to reduce VM downtime.)

The ImageFactory team builds unified images that work on all flavors whenever possible; these provide better test coverage and more convenience for the customer.

Security

We recommend all customers to use our public images -- this way, they benefit from the experience that the image factory team has in creating and managing operating systems for many years. The benefit from the security work we have done, have a stream of updated images available and benefit from the fact that we adapt and enhance images to work on all flavors. We suggest customers to use cloud-init (user-data) and possibly config-management tools to customize images at boot time rather than building their own. If own images need to be built, we recommend to base them on our public images.

OTC public images are good for security, because

  • The build environment is well encapsulated and protected in a dedicated tenant in OTC.
  • The fully automated build process generates reliable and reproducible images from the distributors repositories and the published version-managed configuration. All software comes from RPMs/DEBs.
  • The integrity and authenticity of the packages is cryptographically verified -- the image factory has a list of trusted keys from the distribution vendors.
  • The published images are cryptographically signed.
  • The build process runs nightly from the repository mirrors which are updated several times per day. All security and maintenance updates are thus part of the images.
  • We register _latest images for the distributions at least once a month to support customers regularly rebasing their work on the current patch level of the operating systems. (We could easily do more often should that be required from customers; we recommend passing package_update: true to cloud-init for such scenarios as well.)
  • The images have the repository mirrors registered as update sources, so customers can easily stay up to date with security updates by issueing zypper update, yum update or apt-get update && apt-get upgrade. The repository mirrors are reachabe even without (outgoing) internet access from VMs in OTC, as they are in the provider network (100.64/10).
  • The images are relatively small to reduce the set of attackable services; sshd is the only network facing service enabled by default.
  • The ssh config has been hardened to only allow state-of-the-art cryptography, and to disallow remote root logins and password autentication.
  • The default password for the user linux is RANDOM and while it is displayed from the login prompt in the local noVNC web console emergency access, it is not readable by local users of the machine.
  • The images include hardened network settings (sysctl), where redirects and other potentially risky network settings (e.g. icmp broadcasts) are disabled by default.
  • The images are automatically tested in a sandbox environment after being built; the tests do assure not just generic quality requirements, but also include the validation of the ssh settings.
  • Good random numbers are a requirement for most cryptographic algorithms; good entropy sources are traditionally scarce in virtualized environments. In OTC this is not a problem; beyond the entropy that gets injected at boot time (via cloud-init meta-data), all CPUs in OTC have a hardware random number generator which gets used by the rng-tools installed in our public images.
  • The images are also preconfigured to use a local NTP time source (ntp01/02.otc-service.com as resolved from our standard nameserver at 100.125.4.25) to have a correct system time -- important for checking the validity of certificates.

Reproducing images

In the download directories, you will also find tarballs with the configurations used to build the images. (These tarballs are also cryptographically signed.) You can reproduce the images yourself. For the kiwi images it's most easily done on an openSUSE42 VM -- you need to install kiwi, grub (and optionally zerofree) to do image builds. (For SLES11 images, you need to use a SLES11 VM instead, as there is an older RPM DB format.) Use the create_appliance.sh script to start the build process (it's just a warpper around kiwi).

If you rebuild images on machines outside of OTC, you will need to replace the mirrored package repositories in config.xml with your own local mirrors or the upstream repositories available via the internet.

EulerOS images

EulerOS-2 is a RHEL/CentOS-7 clone, optimized for usage in OpenTelekomCloud. Is is created, maintained and supported by Huawei and built by T-Systems in our image factory along with the other public images as described above. We also have packages, updates and sources mirrored so this can be used conveniently and securely.

Please find more information on the EulerOS pages.

Community and Huawei images

Community images can be identified by the Community_ prefix. Currently we have Ubuntu images in this category. Note that these images do work, but they do not fulfill all the standards outlined above.

Ubuntu image

The Ubuntu image has been built by Canonical for OTC. It has cloud-init set up in a similar way as the ImageFactory images, but differs in a number of ways:

  • It needs a huge 12GB root disk (despite not really requiring more space)
  • It has the ubuntu default user and not linux
  • otc-tools and uvp-monitor are not preinstalled; you need to go to Open Build Service for 14.04 or 16.04. to grab .deb packages or register it as a repository.

Despite building clean .deb packages for uvp-monitor and working with Canonical and paying royalties to them for Ubuntu VM usage, the quality of the Ubuntu image is unfortunately lower than that of the ImageFactory images.

If you have a choice, we advise against using Ubuntu at this point. Take CentOS7 or openSUSE42 as alternatives. They are well maintained, have working CLI tools for OpenStack/OTC, have a working uvp-monitor, extra drivers for SRIOV networking, preconfigured NTP and the standard hardening applied.

Vendor images

We are open to work with vendors to have images coming directly from OS vendors ...

Open Build Service

For additional software, such as drivers and tools, we leverage the Open Build Service from the openSUSE project which offers a very convenient way to provide automated builds and rebuilds for RPM and DEB packages from sources across distributions and architectures. We leverage the home:garloff:OTC project there. Package repostiories are registered in our images (so updates can be retrieved conveniently). The package repositories are signed by this signing key:

pub   2048R/CC6D4077 2016-02-22 [expires: 2020-07-11]
      Key fingerprint = 8A20 8CFD 29C5 0416 8689  A631 84F7 BE81 CC6D 4077
uid       [  full  ] home:garloff:OTC OBS Project

Note: If you are using an old image, you might have an old copy of this key with expiration at the beginning of May 2018. If so, grab the new key from here or from public PGP keyservers and import it into your pacakge management system.

gpg2 --keyserver pool.sks-keyservers.net --recv-keys CC6D4077 | rpm --import
apt-key adv --keyserver pool.sks-keyservers.net --recv-keys CC6D4077

Cloud native apps (some useful links)