Enterprise data centre infrastructure has not changed drastically in the past decade or two, but its use has. Cloud services have changed expectations for how easy it should be to provision and manage resources and also that organizations need only pay for the help they are using.
With the right tools, enterprise data centers could become leaner and more fluid as organizations balance their internal infrastructure against cloud resources to gain the optimal balance. To some extent, this is already happening, as previously documented by Computer Weekly.
The adoption of cloud computing has, of course, been growing for at least a decade. According to figures from IDC, worldwide spending on computing and storage for cloud infrastructure increased by 12.5% year-on-year for the first quarter of 2021 to $15.1bn. In the same period, investments in non-cloud infrastructure increased by 6.3% to $13.5bn.
Although the first figure is spending by cloud providers on their infrastructure, this is driven by demand for cloud services from enterprise customers. Looking ahead, IDC expects spending on compute and storage cloud infrastructure to reach $112.9bn in 2025, accounting for 66% of the total, while spending on non-cloud infrastructure is expected to be $57.9bn.
This shows that demand for the cloud is outpacing that for non-cloud infrastructure, but few experts now believe that the cloud will entirely replace on-premise infrastructure. Instead, organizations are increasingly likely to keep a core set of mission-critical services operating on infrastructure that they control, with the cloud used for less sensitive workloads or where extra resources are required.
More flexible IT and management tools allow enterprises to treat cloud resources and on-premise IT as interchangeable to a certain degree.
Modern IT is much more flexible.
“On-site IT has evolved just as quickly as cloud services have evolved,” says Tony Lock, distinguished analyst at Freeform Dynamics. In the past, it was pretty static, with infrastructure dedicated to specific applications, he adds. “That’s changed enormously in the last ten years, so it’s now much easier to expand many IT platforms than before.
“You don’t have to take them down for a weekend to install new hardware physically – it can be that you simply roll in new hardware to your datacentre, plug it, and it will work.”
Other things that have changed inside the data center are how users can move applications between physical servers with virtualization, so there is more application portability. And, to a degree, software-defined networking makes that much more feasible than five or ten years ago, says Lock.
The rapid evolution of automation tools that can handle both on-site and cloud resources also means that treating both as a single resource pool has become more of a reality.
In June, HashiCorp announced that its Terraform tool for managing infrastructure had reached version 1.0, which means the product’s technical architecture is mature and stable enough for production use – although the platform has already been used operationally for some time by many customers.
Terraform is an infrastructure-as-code tool that allows users to build infrastructure using declarative configuration files that describe what the infrastructure should look like. These are influential blueprints that will enable the infrastructure for a specific application or service to be provisioned by Terraform reliably, again and again.
It can also automate complex changes to the infrastructure with minimal human interaction, requiring only an update to the configuration files. The key is that Terraform can manage an internal infrastructure and resources across multiple cloud providers, including Amazon Web Services (AWS), Azure, and Google Cloud Platform.
And because Terraform configurations are cloud-agnostic, they can define the same application environment on any cloud, making it easier to move or replicate an application if required.
“Infrastructure as code is a nice idea,” says Lock. “But again, that’s something that’s maturing, but it’s maturing from a much more juvenile state. But it’s linked into this whole question of automation, and IT is automating more and more, so IT professionals can focus on the more important and potentially higher-value business elements rather than some of the more mundane, routine, repetitive stuff that your software can do just as well for you.”
Storage goes cloud-native
Enterprise storage is also becoming more flexible in software-defined storage systems designed to operate on clusters of standard servers rather than on proprietary hardware. In the past, applications were often tied to fixed storage area networks. Software-defined storage has the advantage of being able to scale out more efficiently, typically by simply adding more nodes to the storage cluster.
Because it is software-defined, this storage system is also more accessible to provision and manage through application programming interfaces (APIs) or an infrastructure-as-code tool such as Terraform.
One example of how sophisticated and flexible software-defined storage has become is WekaIO and its Limitless Data Platform, deployed in many high-performance computing (HPC) projects. The WekaIO platform presents a unified namespace to applications deployed on dedicated storage servers or in the cloud.
This allows for bursting to the cloud, as organizations can push data from their on-premise cluster to the public cloud and provision a Weka cluster there. Any file-based application can be run in the cloud without modification, according to WekaIO.
One notable feature of the WekaIO system is that it allows for a snapshot of the entire environment – including all the data and metadata associated with the file system – which can then be pushed to an object store, including Amazon’s S3 cloud storage.
This makes it possible for an organization to build and use a storage system for a particular project, then snapshot it and park that snapshot in the cloud once the project is complete, freeing up the infrastructure hosting the file system for something else. If the project needs to be restarted, the snapshot can be retrieved, and the file system recreated exactly as it was, says WekaIO.
But one fly in the ointment with this scenario is the potential cost – not of storing the data in the cloud, but of accessing it if you need it again. This is because of so-called egress fees charged by major cloud providers such as AWS.
“Some cloud platforms look extremely cheap just in terms of their pure storage costs,” says Lock. “But many of them have pretty high egress charges. If you want to get that data out to look at and work on, it costs you a lot of money. It doesn’t cost you much to keep it there, but if you want to look at it and use it, it gets expensive quickly.
“Some people will offer you an active archive where there aren’t any egress charges, but you pay more for it operationally.”
One cloud storage provider that has bucked convention in this way is Wasabi Technologies, which offers customers different ways of paying for storage, including a flat monthly fee per terabyte.
Managing it all
With IT infrastructure becoming more fluid and more flexible and adaptable, organizations may find they no longer need to keep expanding their data center capacity as they would have done in the past. With the proper management and automation tools, enterprises should be able to manage their infrastructure more dynamically and efficiently, repurposing their on-premise IT for the next challenge in hand and using cloud services to extend those resources where necessary.
To make this practical, one area that may have to improve is identifying where the problem lies if a failure occurs or an application operates slowly, which can be difficult in a complex distributed system. This is already a known issue for organizations adopting a microservices architecture. New techniques involving machine learning may help here, says Lock.
“Monitoring has become much better, but then the question becomes: how do you see what’s important in the telemetry?” he says. “And that’s something that machine learning is beginning to apply more and more to. It’s one of the holy grails of IT, root cause analysis, and machine learning makes it much simpler.”
Another potential issue with this scenario concerns data governance, as ensuring that as workloads are moved from place to place, the security and data governance policies associated with the data also travel along with it and continue to be applied.
“If you potentially can move all of this stuff around, how do you keep good data governance on it so that you’re only running the right things in the right place with the right security?” says Lock.
Fortunately, some tools exist to address this issue, such as the open-source Apache Atlas project, described as one-stop data governance and metadata management solution. Atlas was developed for Hadoop-based data ecosystems but can be integrated into other environments.
For enterprises, it looks like the long-promised dream of mixing and matching their IT with cloud resources and being able to dial things in and out as they please may be moving closer.
