Which Version Control Systems Are supported by denodo Virtualization 7.0?

Using Version Control is a denodo Virtual DataPort (VDP) recommended best practice. Version 7.0 of denodo virtualization supports three Version Control Systems (VCS):

  • Microsoft Team Foundation Server (TFS) 2010 or later
  • Apache Subversion (1.7), and
  • Git

Related References:

Denodo Data Virtualization Project Roles

A Denodo virtualization project typically classifies the project duties of the primary implementation team into four Primary roles.

Denodo Data Virtualization Project Roles

  • Data Virtualization Architect
  • Denodo Platform Administrator
  • Data Virtualization Developer
  • Denodo Platform Java Programmer
  • Data Virtualization Internal Support Team

Role To Project Team Member Alignment

While the denodo project is grouped into security permissions and a set of duties, it is import to note that the assignment of the roles can be very dynamic as to their assignment among project team members.  Which team member who performs a given role can change the lifecycle of a denodo project.  One team member may hold more than one role at any given time or acquire or lose roles based on the needs of the project.

Denodo virtualization Project Roles Duties

Data Virtualization Architect

The knowledge, responsibilities, and duties of a denodo data virtualization architect, include:

  • A Deep understanding of denodo security features and data governance
  • Define and document5 best practices for users, roles, and security permissions.
  • Have a strong understanding of enterprise data/information assets
  • Defines data virtualization architecture and deployments
  • Guides the definition and documentation of the virtual data model, including, delivery modes, data sources, data combination, and transformations

Denodo Platform Administrator

The knowledge, responsibilities, and duties of a Denodo Platform Administrator, Include:

  • Denodo Platform Installation and maintenance, such as,
    • Installs denodo platform servers
    • Defines denodo platform update and upgrade policies
    • Creates, edits, and removes environments, clusters, and servs
    • Manages denodo licenses
    • Defines denodo platform backup policies
    • Defines procedures for artifact promotion between environments
  • Denodo platform configuration and management, such as,
    • Configures denodo platform server ports
    • Platform memory configuration and Java Virtual Machine (VM) options
    • Set the maximum number of concurrent requests
    • Set up database configuration
      • Specific cache server
      • Authentication configuration for users connecting to denodo platform (e.g., LDAP)
      • Secures (SSL) communications connections of denodo components
      • Provides connectivity credentials details for clients tools/applications (JDBC, ODBC,,,etc.)
      • Configuration of resources.
    • Setup Version Control System (VCS) configuration for denodo
    • Creates new Virtual Databases
    • Create Users, roles, and assigns privileges/roles.
    • Execute diagnostics and monitoring operations, analyzes logs and identifies potentials issues
    • Manages load balances variables

Data Virtualization Developer

The Data Virtualization Developer role is divided into the following sub-roles:

  • Data Engineer
  • Business Developer
  • Application Developer

the knowledge, responsibilities, and duties of a Denodo Data Virtualization Developer, by sub-role, Include:

Data Engineer

The denodo data engineer’s duties include:

  • Implements the virtual data model construction view by
    • Importing data sources and creating base views, and
    • Creating derived views applying combinations and transformations to the datasets
  • Writes documentation, defines testing to eliminate development errors before code promotion to other environments

Business Developer

The denodo business developer’s duties include:

  • Creates business vies for a specific business area from derived and/or interface views
  • Implements data services delivery
  • Writes documentation

Application Developer

The denodo application developer’s duties include:

  • Creates reporting vies from business views for reports and or datasets frequently consumed by users
  • Writes documentation

Denodo Platform Java Programmer

The Denodo Platform Java Programmer role is an optional, specialized, role, which:

  • Creates custom denodo components, such as data sources, stored procedures, and VDP/iTPilot functions.
  • Implements custom filters in data routines
  • Tests and debugs any custom components using Denodo4e

Data Virtualization Internal Support Team

The denodo data virtualization internal support team’s duties include

  • Access to and knowledge of the use and trouble of developed solutions
  • Tools and procedures to manage and support project users and developers

Denodo Virtual Dataport (VDP) naming Convention Guidance

Denodo provides some general Virtual Dataport naming convention recommendations and guidance.  First, there is the general guidance for basic Virtual Dataport object types and, secondly, more detail naming guidance recommends.      

Denodo Basic Virtual Dataport (VDP) Object Prefix Recommendations

  • Associations Prefix: a_{name}
  • Base Views Prefix: bv_{name}
  • Data Sources Prefix: ds_{name}
  • Integration View Prefix: iv_{name}
  • JMS Listeners Prefix: jms_{name}
  • Interfaces Prefix: i_{name}
  • Web Service Prefix: ws_{name}

Virtual Dataport (VDP) High-Level Project Structure

Different layers are identified when creating logical folders hierarchies within each Data Virtualization project.  The recommended high-Level project folders are:

Connectivity

  • Connectivity, where related physical systems, data sources, and base views are part of this folder.

Integration

  • Integration views include the combinations and transformations views for the next layers. Not directly consumed views at this level.

Business Entities

  • Business Entities include Canonical business entities exposed to all users.

Report Views

  • Report Views include Pre-built reports and analysis frequently consumed by users.

Data Services

  • Data Services include web services for publishing views from other levels. Can contain views need for data formatting and manipulation.

Associations

  • This folder stores associations.

JMS listeners

  • This folder stores JMS listeners

Stored procedures

  • This folder stores custom stored procedures developed using the VDP API.

Denodo Knowledge Base VDP Naming Conventions

Additional more detailed naming convention and Virtual Dataport organization guidance are available in the donodo Community Knowledge Base, under Operations

Knowledge Base Virtual Dataport (VDP) Naming Conventions Online Page

Virtual Dataport (VDP) Naming Conventions Downloadable PDF

Management principles – You can’t Manage what you don’t Measure

Management and Measurement

You can’t manage what you don’t measure is an old management adage that has been used for many years and while most attribute it to Peter Drucker, some claim that the quote was first used by Dr. W. Edwards Deming, although it is a bone of contention whether or not the quote is used in the correct context.

Irrespective of who said it first, I have always agreed with the principle. Coming from a corporate background where this is one of the management principles often used, I was surprised to learn that there are those that strongly disagree with the statement. This group argues that there are many things being managed at work that aren’t measurable, from the confidence we instill in a new, young manager, to the quality of new hires.

The argument is made that quantity is easy to measure, i.e., how much salespeople sell, how many leads marketing creates, or how many phone calls telemarketing makes, but that quality can’t be measured, i.e., excellent customer service, good technical support, or what differentiates a good consultant from a great one.

What to measure

Many organizations use Key Performance Indicators (KPIs) at multiple levels to measure their success at reaching targets, and will then manage the factors influencing the KPI to get it to where they want it to be. A KPI is a value that is measured and shows how effective a company is in reaching key business goals.

Setting a KPI and measuring a specific value is however not always as straightforward as it might seem. To set a KPI, the underlying business objective needs to be properly understood. In one example, a department manager’s KPI included the volume of sales, measured in dollars. In an effort to improve sales, the manager decided to change the remuneration of her sales reps from a fixed salary to a small, basic salary plus commission on sales made. The idea behind this was to incentivize the work, which would lead to increased sales. In the early months after implementing the change, the sales made by account reps did indeed increase dramatically. The CFO then however discovered that the profit margin on those increased sales was substantially lower than the minimum the company expected. The sales reps were discounting the product to increase sales, resulting in a high commission, but the net effect was that the company made less profit.

It is critical that the company’s objectives are clearly understood by all parties and that a suitable metric is measured to check if the objective is being met.

Can quality be measured?

Those arguing that quality, such as excellent customer service, or good technical support can’t be measured, often express the view that the only way that a company can determine how good their service or support is, is by asking the customer. I agree with that statement, but when you do that, aren’t you measuring these aspects? If 50% of your customers feel that your service and support is good, that is a measure against which you can manage and improve those objectives.

The same can be done for any qualitative metric. It merely becomes a question of what is appropriate to measure, and how to obtain those metrics. Qualitative measures often have to be done indirectly, i.e., you need to measure indirect results rather than direct ones.

 The role of Business Intelligence

With the sheer volume of data available across the business, and with much of it residing in different systems, it becomes very difficult to extract the relevant metrics to measure and improve. This is where Business intelligence or BI comes in.

BI utilizes computer-based techniques to spot, extract, and analyze business data, including things like sales, marketing, and production in order to make substantial improvements. Business Intelligence uses data already collected in the business. It is able to utilize data from such diverse sources as website analytics, accounting systems, customer relationship management (CRM) and email management systems.

A Business Intelligence system can automatically use and analyze all the information from these applications in real-time. This enables companies to quickly see, manage and improve their performance. BI goes further than simply measuring performance so that it can be improved, but also helps identify weaknesses in the company.

When an organization grows to the point where huge volumes of data are involved, analytics are used to examine large and varied data sets to uncover correlations, hidden patterns, customer preferences, and market trends; so, organizations can make more-informed business decisions.

Both BI and big data analytics can hugely benefit Organization & Planning within any business. If you have all this information, irrespective of how exactly it was obtained or measured, managing the direction you want to go becomes an informed decision that can be planned for, rather than a guessing game based on ‘gut feel.’

A crucial element that is required in today’s fast-moving world is an organization’s ability to respond rapidly to changes in both the external and internal environment. This is known as Business agility, and it is not possible to do if the business does not measure what is going on inside and around it, and then manages accordingly.

Related References

Data Modeling – Column Data Classification

Data Modeling, Column Data Classification, Field Data Classification
Data Modeling

Column Data Classification

When analyzing individual column data, at its most foundational level, column data can be classified by their fundamental use/characteristics.  Granted, when you start rolling up the structure into multiple columns, table structure and table relationship, then other classifications/behaviors, such as keys (primary and foreign), indexes, and distribution come into play.  However, many times when working with existing data sets it is essential to understand the nature the existing data to begin the modeling and information governance process.

Column Data Classification

Generally, individual columns can be classified into the classifications:

  • Identifier — A column/field which is unique to a row and/or can identify related data (e.g., Person ID, National identifier, ). Basically, think primary key and/or foreign key.
  • Indicator — A column/field, often called a Flag, that has a binary condition (e.g., True or False, Yes or No, Female or Male, Active or Inactive). Frequently used to identify compliance with complex with a specific business rule.
  • Code — A column/field that has a distinct and defined set of values, often abbreviated (e.g., State Code, Currency Code)
  • Temporal — A column/field that contains some type date, timestamp, time, interval, or numeric duration data
  • Quantity — A column/field that contains a numeric value (decimals, integers, etc.) and is not classified as an Identifier or Code (e.g., Price, Amount, Asset Value, Count)
  • Text — A column/field that contains alphanumeric values, possibly long text, and is not classified as an Identifier or Code (e.g., Name, Address, Long Description, Short Description)
  • Large Object (LOB)– A column/field that contains data traditional long text fields or binary data like graphics. The large objects can be broadly classified as Character Large Objects (CLOBs), Binary Large Objects (BLOBs), and Double-Byte Character Large Object (DBCLOB or NCLOB).

Related References

What is a Common Data Model (CDM)?

Data Model, Common Data Model, CDM, What is a Common Data Model (CDM)
Data Model

What is a Common Data Model (CDM)?

A Common Data Model (CDM) is a share data structure designed to provide well-formed and standardized data structures within an industry (e.g. medical, Insurance, etc.) or business channel (e.g. Human resource management, Asset Management, etc.), which can be applied to provide organizations a consistent unified view of business information.   These common models can be leveraged as accelerators by organizations form the foundation for their information, including SOA interchanges, Mashup, data vitalization, Enterprise Data Model (EDM), business intelligence (BI), and/or to standardize their data models to improve meta data management and data integration practices.

Related references

IBM, IBM Analytics

IBM Analytics, Technology, Database Management, Data Warehousing, Industry Models

github.com

Observational Health Data Sciences and Informatics (OHDSI)/Common Data Model

Oracle

Oracle Technology Network, Database, More Key Features, Utilities Data Model

Oracle

Industries, Communications, Service Providers, Products, Data Mode, Oracle Communications Data Model

Oracle

Oracle Technology Network, Database, More Key Features, Airline data Model

Data Modeling – Fact Table Effective Practices

Database Table
Database Table

Here are a few guidelines for modeling and designing fact tables.

Fact Table Effective Practices

  • The table naming convention should identify it as a fact table. For example:
    • Suffix Pattern:
      • <<TableName>>_Fact
      • <<TableName>>_F
    • Prefix Pattern:
      • FACT_<TableName>>
      • F_<TableName>>
    • Must contain a temporal dimension surrogate key (e.g. date dimension)
    • Measures should be nullable – this has an impact on aggregate functions (SUM, COUNT, MIN, MAX, and AVG, etc.)
    • Dimension Surrogate keys (srky) should have a foreign key (FK) constraint
    • Do not place the dimension processing in the fact jobs

Related References