Retail data framework - Common architectural elements

Part 2 - Common architectural elements

 In our previous article from this series we introduced a use case around the data framework for retail stores.

The process was laid out how we've approached the use case and how portfolio solutions are the base for researching a generic architecture. 

The only thing left to cover was the order in which you'll be led through the details.

This article starts the real journey at the very top, with a generic architecture from which we'll discuss the common architectural elements one by one.

This will start our journey into the logical elements that make up the real-time stock control architecture.

Architecture review

As mentioned before, the architectural details covered here are base on real solutions using open source technologies. The example scenario presented here is a generic common architecture that was uncovered researching those solutions. It's our intent to provide guidance and not deep technical details.

This section covers the visual representations as presented, but it's expected that they'll be evolving based on future research. There are many ways to represent each element in this architecture, but we've chosen a format that we hope makes it easy to absorb. Feel free to post comments at the bottom of this post, or contact us directly with your feedback.

From specific to generic

Before diving in to the common elements, it might be nice to understand that this is not a catch all for every possible solution. It's a collection of identified elements that we've uncovered in multiple customer implementations. These elements presented here are then the generic common architectural elements that we've identified and collected in to the generic architecture. 

It's our intent to provide an example for guidance and not deep technical details. You're smart enough to figure out wiring integration points in your own architectures. You're capable of slotting in the technologies and components you've committed to in the past where applicable.  It's our job here to describe the architecture generic components and outline a few specific cases with visual diagrams so that you're able to make the right decisions from the start of your own projects.

Another challenge has been how to visually represent the architectural. There are many ways to represent each element, but we've chosen some icons, text and colours that we hope are going to make it all easy to absorb.

Now let's take a quick tour of the generic architecture and outline the common elements uncovered in my research.

Data integration platform

The logical view splits this solution space into several identifiable platforms where the retail data framework is managed. It takes all three of these platforms to ensure that the data is collected, integrated with the various architectures external to the framework, processed, validated, stored, data science is applied for insights, and exposed back out to the entire retail organisation.

The first we'll look at is the data integration platform where the main action takes place with the retail data framework. Here there are integration microservices and data integration microservices to provide integration with the core platform, data science platform, and storage services. 

Another important set of elements found in this platform are messaging and event processing. Both are essential elements to ensure microservice communications and message transformation within the architecture. To help with data performance, availability, and management there are data caching microservices and data virtualisation microservices. 

Next up, process automation is used to capture processes within the retail organisation, manage the processing and validation of data in a structured traceable manner. The business automation microservices capture all of these processes and ensure proper monitoring of the compliance and regulatory rules for the entire retail organisation. 

Finally, ensuring that the fronting web applications have good visual data representations requires that all microservices are only accessed by authenticated and authorised parties. This is taken care of through the use of an API management element. 

You can sense that this data integration platform contains the elements focusing on microservice deployments which lend themselves to a cloud-native development process using containers and container platforms. 

Core platform

A core platform focusing on security and compliance requires the hosting of retail organisation wide tooling. These are not specifically called out and can be any number of core services or systems hosted within the retail organisation or outside in the form of Software as a Service (SaaS) solutions.

This platform hosts a set of four elements that each support the organisation, starting with compliance and regulatory tooling. This is not the rules mentioned in the previous section, but the tooling backing the development, deployment, and maintenance of the compliance and regulatory rules. 

There should be some form of auditing tooling and governance tooling used to ensure data and the services used to support the retail data framework are properly monitored in their application.

Finally, the authentication and authorisation tooling is the central system used to plug in organisational wide access to the right parties within the retail data framework.

Data science platform

Any retail organisation working in their markets has a vast interest in the behaviour patterns of their customers. At the very least they are using the most basic data science elements, and in advanced cases, they are leveraging all forms of data science to advance their market positions.

In the data science platform we find the more classical business intelligence tooling, often an externally hosted system noted here with a private cloud icon. Providing views and cuts of the mass data collected in retail organisations is the fundamental function of this element. 

The advanced use of not just analytics on their data, but applying more sophisticated technologies like data science (AI / ML) allows retail organisations to gain advantageous insights into customers, trends, products, sales, and other retail activities that raw data analysis can't provide.

Finally, there is data visualisation tooling that provides clear visibility to the consumers of the data and analysis generated from the other elements on this platform.

Storage services

The storage services uncovered in this solution space was a fairly diverse and more high level than the usually noted storage elements found in our architecture. As these storage needs are data focused and organisation wide solutions, you find all the major technologies in the data world applied here, such as data lakes, data warehousing, data hubs, and data marts.

What's next

This was just a short overview of the common generic elements that make up our architecture for the retail data framework use case. 

An overview of this series on retail data framework portfolio architecture:

1. An architectural introduction
2. Common architectural elements
3. Example data architecture

Catch up on any past articles you missed by following any published links above.

Next in this series, taking a look at an example data framework architecture.

(Article co-authored by Iain Boyle, Chief Architect Retail, Red Hat)