Using Logical Data Lakes

Today, data-driven decision making is at the center of all things. The emergence of data science and machine learning has further reinforced the importance of data as the most critical commodity in today’s world. From FAAMG (the biggest five tech companies: Facebook, Amazon, Apple, Microsoft, and Google) to governments and non-profits, everyone is busy leveraging the power of data to achieve final goals. Unfortunately, this growing demand for data has exposed the inefficiency of the current systems to support the ever-growing data needs. This inefficiency is what led to the evolution of what we today know as Logical Data Lakes.

What Is a Logical Data Lake?

In simple words, a data lake is a data repository that is capable of storing any data in its original format. As opposed to traditional data sources that use the ETL (Extract, Transform, and Load) strategy, data lakes work on the ELT (Extract, Load, and Transform) strategy. This means data does not have to be first transformed and then loaded, which essentially translates into reduced time and efforts. Logical data lakes have captured the attention of millions as they do away with the need to integrate data from different data repositories. Thus, with this open access to data, companies can now begin to draw correlations between separate data entities and use this exercise to their advantage.

Primary Use Case Scenarios of Data Lakes

Logical data lakes are a relatively new concept, and thus, readers can benefit from some knowledge of how logical data lakes can be used in real-life scenarios.

To conduct Experimental Analysis of Data:

  • Logical data lakes can play an essential role in the experimental analysis of data to establish its value. Since data lakes work on the ELT strategy, they grant deftness and speed to processes during such experiments.

To store and analyze IoT Data:

  • Logical data lakes can efficiently store the Internet of Things type of data. Data lakes are capable of storing both relational as well as non-relational data. Under logical data lakes, it is not mandatory to define the structure or schema of the data stored. Moreover, logical data lakes can run analytics on IoT data and come up with ways to enhance quality and reduce operational cost.

To improve Customer Interaction:

  • Logical data lakes can methodically combine CRM data with social media analytics to give businesses an understanding of customer behavior as well as customer churn and its various causes.

To create a Data Warehouse:

  • Logical data lakes contain raw data. Data warehouses, on the other hand, store structured and filtered data. Creating a data lake is the first step in the process of data warehouse creation. A data lake may also be used to augment a data warehouse.

To support reporting and analytical function:

  • Data lakes can also be used to support the reporting and analytical function in organizations. By storing maximum data in a single repository, logical data lakes make it easier to analyze all data to come up with relevant and valuable findings.

A logical data lake is a comparatively new area of study. However, it can be said with certainty that logical data lakes will revolutionize the traditional data theories.

Related References

Use and Advantages of Apache Derby DB


Developed by Apache Software Foundation, Apache Derby DB is a completely free, open-source relational database system developed purely with Java. It has multiple advantages that make it a popular choice for Java applications requiring small to medium-sized databases.

Reliable and Secure

With over 15 years in development, Derby DB had time to grow, add new and improve on the existing components. Even though it has an extremely small footprint – only 3.5MB of all JAR files – Derby is a full-featured ANSI SQL database, supporting all the latest SQL standards, transactions, and security factors.

The small footprint adds to its versatility and portability – Derby can easily be embedded into Java applications with almost no performance impact. It’s extremely easy to install and configure, requiring almost no administration afterward. Once implemented, there is no need to further modify or set up the database at the end user’s computer. Alongside the embedded framework, Derby can also be used in a more familiar server mode.

All documentation containing different manuals for specific versions of Derby can be found on their official website, at :

Cross-Platform Support

Java is compatible with almost all the different platforms, including Windows, Linux, and MacOS. Since Derby DB is implemented completely in Java, it can be easily transferred without the need for different distribution downloads. It can use all types of Java Virtual Machines as long as they’re properly certified. Apache’s Derby includes the Derby code without any modification to the elemental source code.

Derby supports transactions, which are executed for quick and secure data retrieval from the database as well as referential integrity. Even though the stored procedures are made in Java, in the client/server mode Derby can bind to PHP, Python and Perl programming languages. 

All data is encrypted, with support for database triggers to maintain the integrity of the information. Alongside that, custom made functions can be created with any Java library so the users can manipulate the data however they want. 

Embedded and Server Modes

Derby’s embedded mode is usually recommended as a beginner-friendly option. The main differences are in who manages the database along with how it’s stored. 

When Derby is integrated as a whole and becomes a part of the main program, it acts as a persistent data store and the database is managed through the application. It also runs within the Java Virtual Machine of the application. In this mode, no other user is able to access the database – only the app that it is integrated into. As a result of these limits, the embedded mode is most useful for single-user apps.

If it’s run in server mode, the user starts a Derby network server which is tasked with responding to database requests. Derby runs in a Java Virtual Machine that hosts the server. The database is loaded onto the server, waiting for client applications to connect to it. This is the most typical architecture used by most of the other bigger databases, such as MySQL. Server mode is highly beneficial when more than one user needs to have access to the database across the network.

Downloading Derby

Derby has to be downloaded and extracted from the .zip package before being used. Downloads can be found at the Apache’s official website:

Numerous download options are presented on there, depending on the Java version that the package is going to be used with. 

Using Derby requires having Java Development Kit (JDK) pre-installed on the system and then configuring the environment to use the JDBC driver. Official tutorials can be found at:

Running and Manipulating Derby DB

Interacting with Derby is done through the use of ‘ij’ tool, which is an interactive JDBC scripting program. It can be used for running interactive queries and scripts against a Derby database. The ij tool is run through the command shell.

The initial Derby connection command differs depending on whether it’s going to be run in embedded or server mode.

For a tutorial on how to use the connect commands, check out https://www.vogella.com/tutorials/ApacheDerby/article.html.

Some Useful Derby DB Documentation

Derby Reference Manual‎: ‎

Derby Server and Administration Guide‎: ‎

API Reference‎:

Derby Tools and Utilities Guide‎: ‎

ij Basics

In conclusion, Derby DB is a lightweight yet efficient tool that can be integrated into various types of Java applications with ease.

What is Development Operations (DevOps)?

With modern businesses continually looking for ways to streamline their operations, DevOps has become a common approach to software delivery used by development and operation teams to set up, test, deploy, and assess applications.

To help you understand more about this approach, let’s briefly discuss DevOps.

What is DevOps?

DevOps comes from two words- ‘development and operations.’ It describes a set of IT practices, which seeks to have software developers and operations team work together on the same project in a more collaborative and free-flowing way.

In simple words, this is a culture that promotes cooperation between Development and Operations teams in an organization to ensure faster production in an automated, recurring manner.

The approach aims at breaking down traditional barriers that have existed between these two important teams of the IT department in any organization. When deployed smoothly, this approach can help reduce time and friction that occur when deploying new software applications in an organization.

These efforts lead to quicker development cycles, which ultimately save money and time, and give an organization a competitive edge against its rivals with longer, more ridged development cycles.

DevOps helps to increase the speed with which an organization delivers applications and services to customers, thereby competing favorably and actively in the market.

What Is Needed for DevOps to Be Successful Executed?

For an organization to appeal to customers, it must be agile, lean, and swift to respond to dynamic demands in the market.  For this to happen, all stakeholders in the delivery process have to work together.

Development teams, which focus on designing, developing, delivering, and running the software reliably and quickly, need to work with the operations team, which is tasked with the work of identifying and resolving problems in the software as soon as possible.

By having a common approach across software developers and operation teams, an organization will be able to monitor and analyze holdups and scale as quickly as possible. This way, they will be able to deliver and deploy reliable software in a shorter time.

We hope that our simplified guide has enabled you to understand what DevOps is and why it is important in modern organizations.

There is nothing like writing

Writing and The Written Word

There is nothing like writing to force you to think and get your thoughts straight.

Warren Buffett

View original post

Poem – A Smile

Writing and The Written Word

Let others cheer the winning man,
There’s one I hold worthwhile;
Tis he who does the best he can,
Then loses with a smile.

— anonymous poem, found in The Book o Virtues, ed. by W. J. Bennett

View original post

Quote – your life’s work

Writing and The Written Word

Whatever your life’s work is, do it well.

—Martin Luther King, Jr.

View original post

The Ills of Knowing Too Much

Writing and The Written Word

There is nothing more disenchanting to man than to be shown the springs and mechanism of any art. All our arts and occupations lie wholly on the surface; it is on the surface that we perceive their beauty, fitness, and significance; and to pry below is to be appalled by their emptiness and shocked by the coarseness of the strings and pulleys.

–Robert Louis Stevenson

View original post