From building, deploying to managing the applications with Azure Services,
we at Dotsquares have done it all. With a team of efficient professionals
who’ve got the knowledge of integrating the Microsoft Azure services in the
client’s project, we provide
end-to-end Azure services.
Darwin is an ongoing Enterprise project, with new and ongoing requirements.
Some of the requirements completed so far include getting data from various
data sources where data sources are Marketing Engines e.g. Facebook Ads,
LinkedIn Ads, Google Analytics,
Google Search Ads 360, Pinterest, Salesforce Marketing Cloud, etc. , merge
it and process data according to client needs. The system has multiple web
connectors which ingest data to the system and process
it as and when the data comes and store it in databases which is used for
creating dashboards and other high level analysis tools. Dotsquares has
implemented various Azure services including Azure DevOps
as per client’s needs to Mange, develop, test, monitor and deploy
Platform Data Processing Architecture
The first requirement for this project was to improve the data
processing architecture. The previous architecture had some loops holes
and it required immense production support due to lot of data load
failures. These loops holes included lack of planning
of data processing, lack of triggers and a planned Architecture.
Dotsquares team helped by understanding the requirements by doing brain
storming sessions with key stakeholders, further working on architecture
to improve the data processing framework, finally changed the framework
for data processing initially for 1 data source.
Dotsquares team helped in development of new connectors (Data source)
like LinkedIn, LinkedIn Ads, YouTube, YouTube Ads, Email, and Salesforce
while some of them are in pipeline like Pinterest. Team started with
requirement gathering followed by data
model design to map out any & all required metrics and calculations
behind metrics. Further development of required connector as per the
discussed metrics and model.
Revamp all other connectors into new processing framework.
This requirement focused in giving public access to master database to
Darwin clients. Further with access to master data in secured way,
customers are able to store brand wise data & see detailed reports
on Power BI embedded.
Less data processing time
No Data failures
Important alerts and reports
Time zone-based triggers
New Synchronous and Asynchronous APIs
Managed Incremental load & Data Pipelines
Improved Data model
Azure SQL Database
Tackled Data Concurrency
AAS refresh Activity
New connectors in new data processing framework
Desired metrics, reports and analytics
Integration with Power BI
Public DB access in secured way
Ad hoc Power BI reports
Advanced Role management & RBAC
Import Dataset in Power BI workspace
Azure Services used
Azure Data Factory
Azure Blob Storage
Power BI & Power BI Embedded
Azure SQL Data warehouse
Since this platform in multi layered including User Interface, Azure
Functions, API, ETL, Azure Data Factory and Database, there is a dedicated
Production support team from Dotsquares managing ongoing production
support needs. Some of the production tasks are data load failure, data
gaps, incremental load failures, access token refresh, Data Availability
tickets, Data validation and lot more.