What solution did the customer want to implement?
The customer wanted to implement a Big Data architecture to group the data within a common Azure database and only share the desired information with its subsidiaries. The idea is to control access to the data and allow subsidiaries to create their own dashboards, giving them great freedom of customization.
The system is complex because it is not a single common platform, but one per subsidiary with different deployment environments. For each version, care must be taken to only issue stable and tested versions.
This is a 100% Microsoft Azure cloud technology. The customer is part of a specific program offered by Microsoft, which allows them to have access to technologies that are not yet available on the market. For example, they benefit from Synapse Analytics, an improved version of the SQL Data Warehouse. This is a real advantage because the project started a year ago with the customer and has only been accessible to the public for 5 months. In addition, this program makes available architects and engineers who support the customer in the choice of technologies and ensure monitoring and problem management. However, since we use components that are not yet available to the public, it is common to encounter bugs or operating problems. This is why we organize regular reviews with Microsoft to decide on the state of the project.
How is this architecture organized?
There are different players, the information comes from several data sources that feed the Data Warehouse.There is a team that adapts if necessary and manages integration in the Data Warehouse, another that is responsible for creating dashboards in PowerBI and finally one that is responsible for administering the Azure platform.
What is your role?
I joined the team in July 2020 to take care of data integration. From external data sources, I adapt them if necessary and integrate them into the Data Warehouse. We are focused on Azure, Azure Data factory and Azure DevOps solutions. In more concrete terms, it is code management, ticket management, automatic deployment workflow (continuous integration, pipeline), Powershell scripting and the development of Azure C# functionality.
What is the main advantage of this technology?
All the data from different systems is centralized in a database with unlimited storage and processing capacity because with this new version we use SQL on demand, it is Azure service for which we do not need to manage our resources, they are allocated directly by Microsoft. And the idea is to duplicate this system in all the subsidiaries and the customer’s environments.
