Using Logical Data Lakes

Today, data-driven decision making is at the center of all things. The emergence of data science and machine learning has further reinforced the importance of data as the most critical commodity in today’s world. From FAAMG (the biggest five tech companies: Facebook, Amazon, Apple, Microsoft, and Google) to governments and non-profits, everyone is busy leveraging the power of data to achieve final goals. Unfortunately, this growing demand for data has exposed the inefficiency of the current systems to support the ever-growing data needs. This inefficiency is what led to the evolution of what we today know as Logical Data Lakes.

What Is a Logical Data Lake?

In simple words, a data lake is a data repository that is capable of storing any data in its original format. As opposed to traditional data sources that use the ETL (Extract, Transform, and Load) strategy, data lakes work on the ELT (Extract, Load, and Transform) strategy. This means data does not have to be first transformed and then loaded, which essentially translates into reduced time and efforts. Logical data lakes have captured the attention of millions as they do away with the need to integrate data from different data repositories. Thus, with this open access to data, companies can now begin to draw correlations between separate data entities and use this exercise to their advantage.

Primary Use Case Scenarios of Data Lakes

Logical data lakes are a relatively new concept, and thus, readers can benefit from some knowledge of how logical data lakes can be used in real-life scenarios.

To conduct Experimental Analysis of Data:

  • Logical data lakes can play an essential role in the experimental analysis of data to establish its value. Since data lakes work on the ELT strategy, they grant deftness and speed to processes during such experiments.

To store and analyze IoT Data:

  • Logical data lakes can efficiently store the Internet of Things type of data. Data lakes are capable of storing both relational as well as non-relational data. Under logical data lakes, it is not mandatory to define the structure or schema of the data stored. Moreover, logical data lakes can run analytics on IoT data and come up with ways to enhance quality and reduce operational cost.

To improve Customer Interaction:

  • Logical data lakes can methodically combine CRM data with social media analytics to give businesses an understanding of customer behavior as well as customer churn and its various causes.

To create a Data Warehouse:

  • Logical data lakes contain raw data. Data warehouses, on the other hand, store structured and filtered data. Creating a data lake is the first step in the process of data warehouse creation. A data lake may also be used to augment a data warehouse.

To support reporting and analytical function:

  • Data lakes can also be used to support the reporting and analytical function in organizations. By storing maximum data in a single repository, logical data lakes make it easier to analyze all data to come up with relevant and valuable findings.

A logical data lake is a comparatively new area of study. However, it can be said with certainty that logical data lakes will revolutionize the traditional data theories.

Related References

A 720-Degree View of the Customer

The 360-degree view of the consumer is a well-explored concept, but it is not adequate in the digital age. Every firm, whether it is Google or Amazon, is deploying tools to understand customers in a bid to serve them better. A 360-degree view demanded that a company consults its internal data to segment customers and create marketing strategies. It has become imperative for companies to look outside their channels, to platforms like social media and reviews to gain insight into the motivations of their customers. The 720-degree view of the customer is further discussed below.

What is the 720-degree view of the customer?

A 720-degree view of the customer refers to a three-dimensional understanding of customers, based on deep analytics. It includes information on every customer’s level of influence, buying behavior, needs, and patterns. A 720-degree view will enable retailers to offer relevant products and experiences and to predict future behavior. If done right, this concept should assist retailers leverage on emerging technologies, mobile commerce, social media, and cloud-based services, and analytics to sustain lifelong customer relationships

What Does a 720-Degree View of the Customer Entail?

Every business desires to cut costs, gain an edge over its competitors, and grow their customer base. So how exactly will a 720-degree view of the customer help a firm advance its cause?

Social Media

Social media channels help retailers interact more effectively and deeply with their customers. It offers reliable insights into what customers would appreciate in products, services, and marketing campaigns. Retailers can not only evaluate feedback, but they can also deliver real-time customer service. A business that integrates its services with social media will be able to assess customer behavior through tools like dislikes and likes. Some platforms also enable customers to buy products directly.

Customer Analytics


Customer analytics will construct more detailed customer profiles by integrating different data sources like demographics, transactional data, and location. When this internal data is added to information from external channels like social media, the result is a comprehensive view of the customer’s needs and wants. A firm will subsequently implement more-informed decisions on inventory, supply chain management, pricing, marketing, customer segmentation, and marketing. Analytics further come in handy when monitoring transactions, personalized services, waiting times, website performance.

Mobile Commerce

The modern customer demands convenience and device compatibility. Mobile commerce also accounts for a significant amount of retail sales, and retailers can explore multi-channel shopping experiences. By leveraging a 720-degree view of every customer, firms can provide consumers with the personalized experiences and flexibility they want. Marketing campaigns will also be very targeted as they will be based on the transactional behaviors of customers. Mobile commerce can take the form of mobile applications for secure payment systems, targeted messaging, and push notifications to inform consumers of special offers. The goal should be to provide differentiated shopper analytics.

Cloud

Cloud-based solutions provide real-time data across multiple channels, which illustrates an enhanced of the customer. Real-time analytics influence decision-making in retail and they also harmonize the physical and retail digital environments. The management will be empowered to detect sales trends as transactions take place.

The Importance of the 720-Degree Customer View

Traditional marketers were all about marketing to groups of similar individuals, which is often termed as segmentation. This technique is, however, giving way to the more effective concept of personalized marketing. Marketing is currently channeled through a host of platforms, including social media, affiliate marketing, pay-per-click, and mobile. The modern marketer has to integrate the information from all these sources and match them to a real name and address. Companies can no longer depend on a fragmented view of the customer, as there has to be an emphasis on personalization. A 720-degree customer view can offer benefits like:

Customer Acquisition

Firms can improve customer acquisition by depending on the segment differences revealed from a new database of customer intelligence. Consumer analytics will expose any opportunities to be taken advantage of while external data sources will reveal competitor tactics. There are always segment opportunities in any market, which are best revealed by real-time consumer data.

Cutting Costs

Marketers who rely on enhanced digital data can contribute to cost management in a firm. It takes less investment to serve loyal and satisfied consumers because a firm is directing addressing their needs. Technology can be used to set customized pricing goals and to segment customers effectively.

New Products and Pricing

Real-time data, in addition to third-party information, have a crucial impact on pricing. Only firms with a robust and relevant competitor and customer analytics and data can take advantage of this importance. Marketers with a 720-degree view of the consumer across many channels will be able to utilize opportunities for new products and personalized pricing to support business growth

Advance Customer Engagement

The first 360 degrees include an enterprise-wide and timely view of all consumer interactions with the firm. The other 360 degrees consists of the customer’s relevant online interactions, which supplements the internal data a company holds. The modern customer is making their buying decisions online, and it is where purchasing decisions are influenced. Can you predict a surge in demand before your competitors? A 720-degree view will help you anticipate trends while monitoring the current ones.

720-degree Customer View and Big Data

Firms are always trying to make decision making as accurate as possible, and this is being made more accessible by Big Data and analytics. To deliver customer-centric experiences, businesses require a 720-degree view of every customer collected with the help of in-depth analysis.

Big Data analytical capabilities enable monitoring of after-sales service-associated processes and the effective management of technology for customer satisfaction. A firm invested in being in front of the curve should maintain relevant databases of external and internal data with global smart meters. Designing specific products to various segments is made easier with the use of Big Data analytics. The analytics will also improve asset utilization and fault prediction. Big Data helps a company maintain a clearly-defined roadmap for growth

Conclusion

It is the dream of every enterprise to tap into customer behavior and create a rich profile for each customer. The importance of personalized customer experiences cannot be understated in the digital era. The objective remains to develop products that can be advertised and delivered to customers who want them, via their preferred platforms, and at a lower cost. 

DataStage – Timestamp from Datetime

Timestamp from date is one of those data type conversions, which I occasionally have to do in DataStage but can never seem to remember. So, I thought I would write this quick post to document the data type conversion code, which is easy, once I finally remember how to do it again. 

The TimestampFromDateTime Function

I use the TimestampFromDateTime(%date%,%time%) function to do this data type conversion. I’m sure there are other ways to achieve the result, but I find this method clean and easy to perform. The TimestampFromDateTime(%date%,%time%) function is in the Functions > Date & Time menu.

To populate the function, you need only add your date field on use ’00:00:00’ as your time element

Time Element example

TimestampFromDateTime(<<Date Field Here>>, ’00:00:00′)

Example transformer code With Date Field

TimestampFromDateTime(Lnk_10.POSTED_DATE, ’00:00:00′)

Infosphere Knowledge Resources

IBM Knowledge Center

The IBM Knowledge Center is the IBM product documentation Site designed to help you plan, install, configure, use, tune, monitor, troubleshoot, and maintain the product.

IBM InfoSphere Information Server Version 11.7.0 documentation

The IBM InfoSphere Information Server Version 11.7.0 documentation provides IIS specific documentation about how to install and use InfoSphere Information Server

IBM Support Portal

The IBM Support Portal is IBM’s technical support resource for all IBM products and services including downloads, fixes, drivers, APARs, product documentation, Redbooks, whitepapers and more.

IBM InfoSphere Information Server Content Hub

The IBM InfoSphere Information Server content hub provides additional on Information Server components including documentation, blog entries, videos, and more.

IBM developerWorks

The IBM InfoSphere Information Server content hub provides additional on Information Server components including documentation, blog entries, videos, and more.

Related Reference

DataStage – Timestamp from date

This is one of those data type conversions which I occasionally have to do in DataStage, but can never seem to remember. So, I thought I would write this quick post to document  the data type conversion code, which is really easy, once I finally remember how to do it again. 

The TimestampFromDateTime  Function

I use the TimestampFromDateTime function to do this data type conversion. I’m sure there are other ways to achieve the result, but I find this method clean and easy to perform. The TimestampFromDateTimefunction is in the Functions > Date & Time menu.

To populate the function, you need only add your date field on use ’00:00:00’ as your time element


TimestampFromDateTime Function Format


TimestampFromDateTime(%date%,%time%)


TimestampFromDateTime with Time Element example

TimestampFromDateTime(<<Date Field Here>>, ’00:00:00′)

Example transformer code With Date Field

TimestampFromDateTime(Lnk_10.POSTED_DATE, ’00:00:00′)

Related References

DataStage – Netezza Connector Action Column

Over the years have occasionally use the action column feature, however, the last month or so I have found myself using it quite a lot. This is especially true in relation to the tea set and not just in relation to the change capture stage.

The first thing you need to know is, if you want to prevent getting the ‘no action column found’ notice on the target stage, need to ensure that the action column has been coded to be a single character field char (1). Otherwise, the Netezza connector stage will not recognize your field as an action column.

While most developers will commonly work with the action column feature in relation to the change capture stage, it can also be very useful if you have created a field from one or more inputs to tell you what behavior the row requires. I have found that this approach can be very useful and efficient under the right circumstances.

Example Pattern for Action Column Using Multiple Source Selects
Example Pattern for Action Column Using Multiple Source Selects

Action column configuration example

Action Column Field Type
Action Column Field Type

 Change Code Values Mapping To Action Column

  • Here’s a quick reference table to provide the interpretation of the change type code to the actual one character action column value to which it will need to be interpreted.

Change Code Type

Change Type Code

Action Column Value

Copy (Data Without Changes)

0

No
value for this Change Type

Insert

1

I

Delete

2

D

Update

3

U

Example Transformer Stage, Derivation

  •  Here is a quick transformer stage derivation coding example to take advantage of the action call capabilities. If you haven’t already handled the removal of the copy rows, you may also want to add a constraint.
  • The combination I most frequently find myself using is the insert and update combination.
if Lnk_Out_To_Tfm.change_code=1 then ‘I’

Else if Lnk_Out_To_Tfm.change_code=2 then ‘D’

Else if Lnk_Out_To_Tfm.change_code=3 then ‘U’

Related References

Home > InfoSphere Information Server 11.7.0 > InfoSphere DataStage and QualityStage > Developing parallel jobs > Introduction to InfoSphere DataStage Balanced Optimization > Job design considerations  > Specific considerations for the Netezza connector

End Of Support For IBM InfoSphere 9.1.0

IBM Information Server (IIS)
IBM Information Server (IIS)

End of Support for IBM InfoSphere Information Server 9.1.0

IBM InfoSphere Information Server 9.1.0 will reach End of Support on 2018-09-30.  If you are still on the InfoSphere Information Server (IIS) 9.1.0, I hope you have a plan to migrate to an 11-series version soon.  InfoSphere Information Server (IIS) 11.7 would be worth considering if you don’t already own an 11-series license. InfoSphere Information Server (IIS) 11.7 will allow you to take advantage of the evolving thin client tools and other capabilities in the 2018 release pipeline without needing to perform another upgrade.

Related References

IBM Support, End of support notification: InfoSphere Information Server 9.1.0

IBM Support, Software lifecycle, InfoSphere Information Server 9.1.0

IBM Knowledge Center, Home, InfoSphere Information Server 11.7.0, IBM InfoSphere Information Server Version 11.7.0 documentation