• Tutorial
    • AWS
    • Oracle
    • WordPress
    • Azure DevOps
  • About me
Tutorial
    AWS
    Oracle
    Wordpress
    Azure DevOps
About me
cedricleruth - It's still magic even if you know how it's done
  • Tutorial
    • AWS
    • Oracle
    • WordPress
    • Azure DevOps
  • About me

How to integrate your data into a Cloud ERP software?

Any organization considering acquiring an ERP software module from a CRM to Finance, HR, SCM, or Sales & Marketing will ask how to integrate their existing data. Whether it is a Cloud or On-Premise ERP, the methods for importing, exporting, and continuously synchronizing data within your organization’s existing systems are the same. The main difference to the Cloud is that it is usually required to duplicate some existing organizational data. In the following sections, I’ll share my typical explanation of those notions with new customers.

First of all, a quick glossary:

  • An Enterprise resource planning (ERP) software is a business management software that an organization can use to collect, store, manage, and interpret data from their activities. (Financial Accounting, Management Accounting, Human resources, Manufacturing, Order processing, Supply chain management, Project management, Customer relationship management, etc.)
  • Customer relationship management(CRM) allows a company to manage and analyze its interactions with its past, current, and potential customers.
  •  Cloud computing software is a software model in which customers’ services are available over the internet on a subscription/per-use basis. 
  • A Middleware is a communication software to enable communication and data management between distributed applications.
  • An ETL is a Middleware short for Extract, Transform, and Load. Three functions combined into one tool to pull data out of systems, clean and transform those data, and load them into another software.

Data integration in an ERP Cloud

Data integration is about combining data from different internal and external sources into a single, centralized repository. For example, a business can store customer data in a local database, manage inventory data with a third-party platform, and want to centralize all those data into a data warehouse or an ERP module like a CRM.

Such situations are not uncommon. As a business grows and changes, so do their software and data needs, and a strategy that once made sense needs to be revised.

The ETL process and other modern data streaming approaches are at the heart of data integration. Data integration begins with extracting data from multiple sources and moving them into a single data warehouse. (For businesses and organizations that do not use a data warehouse, the process is similar, although the data will be integrated directly from the source.) To facilitate the integration process, a cloud ERP offers a range of interface points, including REST, SOAP, and a BULK API.

During the transformation step, data is cleaned, validated, organized, and standardized. At this point, all of the different datasets are now in conversation with each other. Finally, the converted data is loaded at its final destination.

Data migration or Data integration?

The terms describe distinct and separate processes. They do, however, share some of the same implementation techniques.

Data integration is combining data from multiple sources, internal and external, into a target system. Data integration describes a unified set of smaller processes. Each process allows the extraction, transformation, and loading of a different data model. (customers, addresses, orders, etc.)

Data migration, involves moving data from one system to another. When a company decides to change its existing CRM system, or when it decides to downgrade from an older version to a more recent one, it must migrate all data from the current software to the new one.

Common integration methods

So far, we’ve provided an overview of the data integration process and how it combines data from multiple origins into one view and source. Some of the different data integration methods include

Manual data consolidation

This part of the process typically requires a conventional ETL, although some companies may use built-in custom tools or a simple excel extraction. Manual consolidation can work well for smaller, more specific datasets that don’t require a deep clean, but it can be too time-consuming and error-prone for more massive datasets. Besides, the lack of real–time data limits its usefulness.

Propagation of data from source applications

The goal here is to propagate the data from the individual applications to the ERP, and the integration logic to achieve this expands in the client applications. Rather than a standard tool or approach to moving data into the warehouse, each application takes responsibility for moving their data to the central store. This method is generally adopted because there can be heavy data cleaning and manipulation, and the application is in the best position to understand and perform these operations.

This approach is challenging to maintain because applications are subject to change, which often means that the integration logic needs to be rebuilt or adjusted.

Propagation of data using a Middleware

This method ignores the logic of application integration and shifts the responsibility to the Middleware. For example, a subscription mechanism configured between the Cloud ERP and the data warehouse ensures that whenever there is an update, an event is triggered to automatically publish the data to the warehouse, keeping it up to date.

Even when applications change, the Middleware maintains his function as a bridge transferring data to the ERP.

For this method to work, there must be an implementation layer that manipulates and transforms the data into a format that the consumer understands.

Data virtualization

In virtualization, data is not extracted and stored in a common repository but provides a mechanism to access data remotely from multiple sources.

The technique has the advantage of not having to create and manage a Middleware and offers up-to-date data in real-time without any data replication. It is perfect for highly secure applications that do not allow data to be stored elsewhere. However, this limits the scope of how ERP can use this data. The ERP is also constantly polling these data sources, adding performance loads to those databases.
This technique is not available on all Cloud ERPs.

The challenges of data integration

54% of Salesforce business customers identified integrating apps and data sources as their top challenge. Let’s take a look at a few factors where data integration remains a challenge:

Find the right experts

Integrating a cloud ERP with a data warehouse requires experts in different fields such as Cloud technologies, ERP modules, data warehouses, and Middleware technologies. Building such a team and ensuring that they communicate effectively can be a challenge.

Complexity of systems

Bringing together data from many systems using different technologies and locations can be a complicated task. The scale, volume, and complexity of this process require substantial planning and coordination.

Data mapping

Because data fields tend to be stored with different names and types in data sources, it isn’t easy to map each lot to the destination system. Some of the data sources could also be existing systems with significant data gaps. Solving these issues requires collaboration between business and technical stakeholders, who profoundly understand the data.

Ensure continuous data integration

Data integration is not a one-time task. The initial effort to import data is significant. Nevertheless, you need constant efforts to update the ERP and data warehouse when changes occur automatically.

Despite these challenges, data integration remains an essential part of an organization’s strategy to achieve a unified data view. Having a clear integration strategy and using a data integration tool overcomes these barriers.

Uniform data integration strategy

Consolidating the mix of Cloud and on-premises sources can mean different approaches to integrating their data. However, divergent paths can lead to inconsistent data processing, which in turn can compromise data quality. Creating a uniform strategy that ensures data integrity and synchronization despite systems’ individuality can be difficult.

How to define your integration strategy?

Identify your stakeholders

These can include executive sponsor, cloud ERP experts, data engineers, customers, and other specialists with a comprehensive organizational data view.

Ask the right questions

What are the budget limits, time, and availability of stakeholders?
Does your data need to be available in real-time, or can it be pulled on-demand or in batches?
What works best for your business: manual consolidation, propagating data to a warehouse using applications, reproducing data to a warehouse using a Middleware, or keeping your data bounded using virtualization?

Match the ERP data fields to yours.

Will you be using APIs, direct database access, Queuing, Streaming to manage the integration?

There is no conventional approach to integrating data into an ERP. Some organization sticks to manual integration while others use application logic, a Middleware, or a hybrid approach.

The final solution an organization achieves depends on many factors: the propensity to create a data warehouse, the availability of resources such as time and money, the size of the data sets, needing the data synchronized in real-time.

Reduce recurring human intervention

A data integration tool helps simplify the integration process’s complexity by providing an automated mechanism that consolidates data from multiple sources on-premises and in the Cloud. Such a tool not only enables faster ETL operations but also ensures continuous and real-time updates of the centralized data store. Doing so minimizes human intervention, reduces errors, saves time, and thus increases productivity and data quality.

Additionally, the tool makes it easier to scale as more data sources are added. Rather than having a fragmented approach with a different integration method for each source, the tool offers a consistent solution.

How to configure your own Dropbox alternative encrypted private cloud storage on AWS with Owncloud?

I fell in love with how valuable a file hosting service can be around 2010 by getting introduced to Dropbox.

Being able to access your files on the go and synchronize them between multiple computers was a game changer at a time where Cloud computing wasn’t yet a buzzword.

1) Why should you use an alternative to Dropbox?

Dropbox sadly was also one of the first Cloud providers to violate our trust. And not by a little, with more than 13 massive data privacy issue. (See detail here: https://en.wikipedia.org/wiki/Criticism_of_Dropbox)

In a world where our private data are worth millions, we should always strive to keep them private.

2) How is Owncloud private cloud storage a good alternative?

So that’s where Owncloud(https://owncloud.org/) come into play.
Owncloud is the most notable open source file Sharing software with more than 25 000 customers.

It’s hosted by you and can be encrypted from the client side so even if someone gets physical access to your host they won’t get access to your data.

3) Is it ok to host your private data on AWS EC2?

I first started by hosting my private cloud on an old laptop at home but quickly decided to host it on an AWS EC2 for three reasons:
– I didn’t want to have a computer running 24h/24h at home with a firewall port open. But if you can do that I’ll strongly recommend it!
– AWS EBS Cold HDD hard drives are cheap and encryptable. (You can run it for free if you are free-tier Eligible https://aws.amazon.com/free/ up to 30Gb)
– Owncloud can be set to be end-to-end encrypted so even if someone would access the EC2 host they would not have access to your files. (https://owncloud.org/gdpr/)

4) How to install your encrypted private file storage on AWS EC2 with Owncloud?

  • Create a free account and connect to your AWS console: https://console.aws.amazon.com/
  • Navigate to EC2 and launch Instance
    • Step 1: Choose AMI: Choose the last Amazon Linux AMI available
    • Step 2: Choose Instance Type: Select the t2.micro size then click next
    • Step 3: Configure Instance Details: Keep default.
    • Step 4: Add Storage:
      • Set the root hard drive as 8Gb Magnetic
      • Add New Volume: EBS /dev/sdb/ Choose the size of the hard drive you’ll need in Ebs Cold HDD (SC1) (Lowest cost HDD volume designed for less frequently accessed workloads) – You can set it as encrypted
    • Step 5: Add Tags(Optional) : Add a Name Key and a Value: MyOwnCloud
    • Step 6: Configure Security Group:
      • Create a new security group name: OwnCloudSecurityGroup
      • Add a rule to have SSH, HTTP, HTTPS from everywhere
    • Review and launch
    • Create a new key pair to be able to access your instance with SSH (and download it)
  • Set an Elastic IP to your instance
    • On the EC2 console navigate to Elastic IP
    • Allocate new address
    • Name it (example OwnCloudElasticIP)
    • Select it and then Action > Association Address > Select your instance > Associate
  • Install Owncloud on your instance through ssh

If you are setting it with a custom domain name, you own and want to setup SSL. Set it up as follow:

  • Configure your owncloud
    • go to your_ip/owncloud
    • Set an admin user password
    • Select storage database
    • Set the storage as /owncloudDisk/
    • Set your clouddbuser as MariaDB DB information

That’s it your Owncloud is ready to use.

Don’t forget to activate encryption in your settings and feel free to comment if something went wrong!

Want to be notified when there is more?

Get in touch:

Categories

  • AWS EC2
  • AWS Lambda
  • AWS S3
  • AWS Tutorials
  • Azure DevOps
  • Highlight
  • IT Architecture
  • Oracle ADF
  • Oracle Database
  • Oracle SOA Suite
  • Oracle Tutorials
  • Route53
  • Tutorials
  • Weblogic
  • Wordpress Tutorials

Read more about:

ADF Apache AWS Bean binding bpel Certificate cli cloud convert Database Deployment EC2 ERROR Exception file First Value GIT HTTPS Icons IllegalStateException iterator Java Jdevelopper lambda Oracle Oracle ADF Performance php programmatically RichTable route53 row S3 script sendRedirectforward soa sql SSL table View Criteria View Object VO weblogic WordPress