A collection of information technology, consulting, and business knowledge
A database is an organized collection of data, generally stored and accessed electronically from a computer system. Where databases are more complex they are often developed using formal design and modeling techniques.
The database management system (DBMS) is the software that interacts with end users, applications, and the database itself to capture and analyze the data. The DBMS software additionally encompasses the core facilities provided to administer the database. The sum total of the database, the DBMS and the associated applications can be referred to as a “database system”. Often the term “database” is also used to loosely refer to any of the DBMS, the database system or an application associated with the database.
Computer scientists may classify database-management systems according to the database models that they support. Relational databases became dominant in the 1980s. These model data as rows and columns in a series of tables, and the vast majority use SQL for writing and querying data. In the 2000s, non-relational databases became popular, referred to as NoSQL because they use different query languages.
As the data modeling process in denodo moves through the conceptual layers of the data warehouse, there is an evolution of the data structure and their associated metadata.
The Base Layer
As the modeling process begins the base layer is the ingestion layer, where the source system data structures are recreated in denodo and field are transformed in denodo Virtual Query Language (VQL) data types. the Business layer is what folks with a traditional data warehousing background would think of as Staging or landing. These base layer views should most closely mirror the technical structure and data characteristics of the input data source and will be the least business friend in their organization, naming, and metadata.
The Semantics layer
The semantics layer is where the major data reorganization, data transformation, and the application of business friend field names and metadata begins. The semantics layer is what folks with a traditional data warehousing background would think of as the Data Warehouse (DW) or Enterprise Data Warehouse (EDW). The semantics layer of the logical data warehouse (LDW) performs serval tasks:
Data from multiple input sources are consolidated
The model becomes multi-dimensional (Fact and Dimension oriented)
Field names and descriptive metadata are changed to meaningful, domain normalized, business-friendly names and descriptions.
Domain normalizing business rules and transformations are applied.
Serves as a data source for the business layer and reporting layer.
The Business Layer
The business layer, which is considered optional by denodo, is modeled along a more narrow business subject orientation and more specialized business rules are applied. This is what folks with a traditional data warehousing background would think of as a Datamart (DM).
The business layer of the logical data warehouse (LDW) performs serval tasks:
Limits and optimizes the data to facilitate business intelligence and report activities concerning a specific line of business or business topic (e.g. Financials, Human Resources, Inventory, Asset management, etc. )
Business-specific/customized rules and metadata are applied
Supplements the semantic layer and serves as a data source for the reporting layer.
Additional data consolidation and data structure denormalization (flattening) may occur in the business layer
The Reporting Layer
The reporting layer, which is considered optional by denodo, is the most customized layer and sees the most reporting topic specialization and specific need transformation. The reporting layer is where a traditional data warehousing may provide customized reporting, or system interface views, interface ETL’s to produce interface files, and reporting team do more of their own development.
The reporting layer of the logical data warehouse (LDW) performs serval tasks:
Provides consumer-specific customized rules and metadata
Provides consumer-specific data organization/layouts
Data is optimized for consumer purposes and may be highly or entirely denormalized to meet consumer needs.
The denodo “Base Layer” in the Logical Data Warehouse (LDW) can be thought of as the Data Staging local layer in a more traditional data warehouse (DW) development pattern. The Base layer is the level at which the source system data structures are transformed into denodo field types and the source data structures are rendered as created in base views (bv).
Base views (bv), the first step in virtualizing data, are the denodo structures reflecting the source system structure and the second step behind the data source connection and, therefore, are essential elements for the other layers of the Logical Data Warehouse (LDW). To provide some guidance to facilitate the usefulness and performance of base views here are some best practices:
Use consistent Object Naming conventions. It is strongly recommended that the denodo standard naming conventions be used.
Import and or create the Primary Keys (PK), Foreign Keys (FK), and Associations.
Have Statistics Collection been set and include all critical fields?
Base views, as a rule, should not be cached unless absolutely necessary for reasons of performance.
Create indexes on Primary Keys (PK), Surrogate keys, and Foreign Keys (FK)
Create performance Indexes to mirror sources system to improve performance.
Populate view Metadata properties describing the type and the nature of the data which the view contains. Note: if you have a governance team, they may want to manage the metadata in the denodo data catalog.
Retain the original table name (applying naming convention prefix and field names to facilitate data Lineage traceability.?
Use denodo tools against tables, where possible, rather than (Manual) SQL views, Database views, or Stored Procedure. Denodo cannot rewrite or optimize these objects.
Field metadata should be annotated with “Not Used” if the field is always null, blank, or empty. This saves time and labor when working with levels and researching data issues
In denodo associations follow the same concept as modeling
tools, which can be described as an ‘on-demand join.’
Should Associations Be Created In the Denodo Model?
You don’t necessarily need to define an Association at every
level; usually, the best practice is to apply associations at the following
On final views published for data consumers,
indicating relationships between related views; Especially, on published web
On the component views below, any derived view
that brings together disparate (dissimilar) data sources. The associations should be defined as
Referential Constraints whenever appropriate to aid the optimization engine.
On the component views below, any derived view
that joins a “Base View from Query” with standard views, since Base
Views from Query cannot be rewritten by the denodo optimization engine. Often Base Views from Query create
These best practices should cover the majority scenarios;
beyond these guidelines, it is best to take an ad-hoc approach to create
Associations when you see a specific performance/optimization.
Are Associations important in Denodo?
In a nutshell, associations performance and the efficiency
of the denodo execution optimizer along with other model metadata, such as:
A coworker recently asked a question as to whether denodo
generated joins automatically from source RDBMS database schema. After searching, a few snippets of
information became obvious. First, that
the subject of inheriting join properties was broader than joins and needed to
in modeling associations (joins on demand). Second, that there were some denodo
design best practices to be considered to optimize associations.
Denodo Automatically Generate Joins From the Source System?
After some research, the short answer is no.
Denodo Inherit Accusations From A Logical Model?
The short answer is yes.
Denodo bridges allow models to be passed to and from other
modeling tools, it is possible to have the association build automatically,
using the top-down approach design approach and importing a model, at the
Interface View level, which is the topmost level of the top-down design
However, below the Interface view level, associations and or joins are created manually by the developer.
Denodo provides some general Virtual Dataport naming
convention recommendations and guidance.
First, there is the general guidance for basic Virtual Dataport object
types and, secondly, more detail naming guidance recommends.