The Department for Business, Energy, and Industrial Strategy (BEIS) replaced an Excel-based system for grant applications with a bespoke data management system (DMS) built by consultancy and software firm Aiimi.
BEIS has signed a one-year contract worth £900,000 to capture and securely process Covid-19 business support grant return data. The system will also be used for a Green Homes Grant.
The department has worked with Milton Keynes-based technology company Aiimi to build a near-real-time system. This will see grant-related data from more than 300 local authorities across the UK captured, ingested, prepared, and enriched using a combination of Microsoft Azure and Databricks technologies.
Gavin Dollin, product sponsor and head of data services at BEIS, said: “The data management system at BEIS lays the foundations for a great data service. It brings together tech from Microsoft Azure and Databricks to create a platform that supports our current needs and will change and grow to match our ever-evolving data journey.”
The Covid-19 business support grants will supply loans, tax relief, cash grants, financial support, and trading grants. The Green Homes Grant encourages homeowners and residential property owners to install energy-efficient improvements by offering financial contributions towards insulation, low-carbon heating, and draft-proofing costs.
Gavin Dollin, BEIS
Ryan Moore, head of data and analytics at Aiimi, said: “The blended BEIS and Aiimi project team has set new standards in how we approach data platform design, applying user research techniques to understand and define the solution’s capabilities. Our joint effort has created a solid grounding for an evolving data platform and a powerful tool in measuring the performance of initiatives designed to drive economic change and recovery across the UK.”
Jack Lawton, data science principal at Miami, has been the project lead on the engagement with BEIS. “We see content and data as two sides of the same coin,” he said. “If you start treating content like data and vice-versa, you get the most value. That’s our approach.”
He said that Aiimi’s main product, Insight Engine, is designed to treat content and data the same way, but it does not feature in the BEIS project.
He said the firm had grown from around 30 people when Lawton joined seven years ago to over 100. It moved from London to Milton Keynes to accommodate the growth but operates a remote-fist policy for recruitment.
The BEIS project team comprised 10-15 Aiimi consultants at different times and a similar number of civil servants.
The project is based on the Microsoft Azure platform. The data lake it has set up is supplemented by a data capture application that concedes to BEIS’s IT security policy.
On the data processing and analysis side, the Databricks technology it uses comes into its own, he said, because the data is not always supplied in the correct format. “We’re dealing with data submitted by local authorities in Excel spreadsheets. Unfortunately, Excel allows a lot of freedom to make changes, which can bring problems when you want to bring in a whole lot of them and somehow unify them,” said Lawton
“We see content and data as two sides of the same coin. You get the most value if you start creating content like data and vice-versa. That’s our approach.”
Jack Lawton, Aiimi
Databricks allows the running of complex analytics at scale using Apache Spark, the open-source technology that the founders of Databricks initially invented. The project team has also used Databricks’ Delta Lake data storage system, the virtue of which is to show the history of what has been done on a dataset and processing it, said Lawton.
In terms of the organizational design of the data management system project team, he said it exemplified the department’s general approach, which is to run it as a blended team. “Who knows BEIS data better than BEIS? They are the subject matter experts. And by developing it with them, we are upskilling the team so they can support the DMS when we’ve finished,” said Lawton.
The role of the DMS is to enable BEIS to better report to the government on the grants applications data gathered by local authorities from individuals and companies. The cleansed and unified data store can also be analyzed using machine learning algorithms or dashboards.
“One of the key elements of the DMS platform is the ‘lab’ and ‘exhibits’ layers in the data lake,” said Lawton. “The purpose of these is to enable analysts to explore and play with the data and show it without too many strings. And then, if that analysis is of value to the business, it is relatively straightforward to adapt it under central management by the DMS team.
“The ability to take data from external sources into the data capture portal is the differentiating benefit of this system for BEIS. You can take sensitive data from local authorities and securely pipeline it in. Everything is virus scanned.”
The data engineering for the project has been completed, as has a limited roll-out of the Covid grant returns, but the entire system is not yet in the hands of local authorities. The contract runs until December 2022.
