How to work with MapReduce Key Value Pairs in Pentaho Data Integration

Pentaho Kettle Parameters and Variables: Tips and Tricks

Parameters and Variables

Definitions upfront

Example

jb_main.kjb

jb_slave.kjb

tr_set_variables.ktr

tr_show_param_values.ktr

Making Parameters available for all subprocesses in an easy fashion

1. In the main job specify the parameters that you want to pass in in the Job Settings:
   This way parameters and their values can be passed in from the command line in example.
2. Right after the Start job entry use the Set Variables job entry. Specify the variable names, reference the parameters you set up in step 1 and set the scope to Valid in the current job.
3. There is no need to specify any parameters/variables in any of the subprocesses.

What if I still want to be able to run my subprocess independently sometimes?

1. In the subprocess, specify the parameter that you want to be able to pass in. In our example (which is based on the previous example), we modified the transformation tr_show_param_values.ktr and added following parameters to the Transformation Settings:
   We also amended the Get Variables step to make use of these parameters:
   This way, we can already run this transformation on its own. Now we only have to adjust the parent job so that we can run it from there as well.
2. In the parent job, in the Job or Transformation job entry settings, go to the Parameters tab and tick Pass all parameter values down to the sub-transformation/sub-job. Next, as the Parameter set the name of the parameter you defined in the subprocess. As the Value define the variable that you want to pass down: ${variable}. This assumes that this variable was set beforehand by some Set Variables job entry/step.
   In our case, we modified transformation job entry in the job jb_slave.kjb and added following mapping to the job entry settings in the Parameters tab:

Closing remarks

Subtitle: Kettle Testing for the Non-Java Developers

Announcing the KTF Beta:

Precautions

The small print upfront

Motivation

Contribute!

Features

• Supports multiple input datasets
• Supports multiple output datasets
• Supports sorting of the output dataset so that a good comparison to the golden output dataset can be made
• The setup is highly configurable (but a certain structure is enforced - outlined below)
• Non conflicting parameter/variable names (all prefixed with “VAR_KTF_”)
• Non intrusive: Just wraps around your existing process files (except some parameters for db connections etc will have to be defined … but probably you have this done already anyways)

Current shortcomings

• Not fully tested (only tested with included samples and on PostgreSQL)
• Currently works only with Table input / output transformations. Text/CSV file input/output will be supported in a future version (which should not be too complicated to add).
• Dictates quite a strict folder structure
• Limited documentation

Project Folder Structure

1. Adjust parameter values in config/.kettle/kettle.properties and JNDI details in config/simple-jndi/jdbc.properties
2. Your transformations have to be prefixed for now with “tr_” and your jobs with “jb_”. Do not use any special characters or spaces in your job/transformation names. Your transformations and jobs have to be saved within repository/main. There is currently now subfolder structure allowed within this folder.
3. Within the config/test-cases directory create a folder for the processes you want to test. A process can be a transformation or a job. Name these folders exactly the same as the job/transformation you want to test (just without the file extension).  Each process folder must have an input and output folder which hold the DDL, Kettle data type definitions and in case of the output the sort order definition (see tr_category_sales sample on how to set this up). If your output dataset does not require any sorting, create an empty sort def file (see tr_fact_sales example). Note that KTF can handle more than one output/input dataset.
4. The process folder must also contain at least one test case folder (which has to have a descriptive name). In the screenshot above it is called “simpletest”. A test case folder must contain an input and output folder which each hold the dedicated datasets for this particular test case. In case of the output folder it will hold the golden output dataset(s) (so that dataset that you want to compare your ETL output results to).
5. Users working on Windows: For all the CSV output steps in transformations under /repository/test set the Format to Windows (Content tab). KTF has not been tested at all on Windows, so you might have to make some other adjustments as well.
6. Add environment variables defined in config/set-project-variables.sh to your .bashrc. Then run: source ~/.bashrc
7. Start Spoon (this setup requires PDI V5) or run the process from the command line.
8. Run the test suite
9. Analyze results in tmp folder. If there is an error file for a particular test case, you can easily visually inspect the differences like this:

Future improvements

• Write test results to dedicated database table
• Improvement of folder structure
• Support for text file input and output for main processes (jobs/transformations)

FAQ

Going Agile: Sqitch Database Change Management

You have your database scripts under a dedicated version control and change management system, right? If not, I recommend doing this now.

While there have been handful of open source projects around which focus on DB script versioning and change management control, none has really gained a big momentum and a lot of them are dormant.

But there is a new player on the ground! A light at the end of the db change management tunnel - so to speak. David Wheeler has been working on Sqitch over the last year and the results are very promising indeed! Currently the github projects shows 7 other contributors, so let’s hope this project gets a strong momentum! Also a new github project for a Sqitch GUI was just founded.

Why I like Sqitch:

• You can run all the commands from the command line and get very good feedback.
• Everything seems quite logical and straightforward: It’s easy to get to know the few main commands  and in a very short amount of time you are familiar with the tool.
• You can use your choice of VCS.
• It works very well.

Supported DBs are currently MySQL, Oracle, SQLite and PostgreSQL. CUBRID support is under way.

So what do we want to achieve? So what do we want to achieve?

Bring all DDL, stored procedures etc under version control. This is what Git is very good for (or your choice of CVS).

Keep track of the (order of) changes we applied to the database, verify that they are valid, be able to revert them back to a specific state if required. Furthermore, we want to deploy these changes (up to a specific state) to our test and production databases. This is was Sqitch is intended for:

The below write-up are my notes partially mixed with David’s ones.

Info

http://sqitch.org/

https://github.com/theory/sqitch

https://metacpan.org/module/sqitchtutorial

Forum:

https://groups.google.com/forum/#!forum/sqitch-users

Installation

Options:

PostgreSQL: cpan App::Sqitch DBD::Pg (You also have to have PostgreSQL server installed)

SQLite: cpan App::Sqitch DBD::SQLite

Oracle: cpan App::Sqitch DBD::Oracle (You also have to have SQL*Plus installed)

MySQL: cpan App::Sqitch

If you want to have support for i.e. PostgreSQL and Oracle you can just run:

PostgreSQL: cpan App::Sqitch DBD::Pg DBD::Oracle

For more install options see here.

Below I will only discuss the setup for PostgreSQL.

On the terminal run:

$ sudo cpan App::Sqitch DBD::Pg

During installation it will ask you for the PostgreSQL version. If you are not sure, run:

$ psql --version

It then asks you for a PostgreSQL bin directory. On Ubuntu, this is located in:

/usr/lib/postgresql/9.1/bin

Next it will ask you where the PostgreSQL include directory is located. You can find this out by running the following:

$ pg_config --includedir

If you don’t have pg_config installed, run first:

$ sudo apt-get install libpq-dev

The include location of Ubuntu should be:

/usr/include/postgresql

Once installation is finished, check out the man page:

$ man sqitch

Within your git project directory, create a dedicated folder:

$ mkdir sqitch

$ git add .

$ cd sqitch

$ sqitch --engine pg init projectname

Let's have a look at sqitch.conf:

$ cat sqitch.conf

Now let’s add the connection details:

$ vi sqitch.conf

uncomment and specify:

[core "pg"]

client = psql

username = postgres

password = postgres

db_name = dwh

host = localhost

port = 5432

# sqitch_schema = sqitch

If psql is not in the path, run:

$ sqitch config --user core.pg.client /opt/local/pgsql/bin/psql

Add your details:

$ sqitch config --user user.name 'Diethard Steiner'

$ sqitch config --user user.email 'diethard.steiner@'

Let’s add some more config options: Define the default db so that we don’t have to type it all the time:

$ sqitch config core.pg.db_name dwh

Let's also make sure that changes are verified after deploying them:

$ sqitch config --bool deploy.verify true

$ sqitch config --bool rebase.verify true

Check details:

cat ~/.sqitch/sqitch.conf

Have a look at the plan file. The plan file defines the execution order of the changes:

$ cat sqitch.plan

$ git add .

$ git commit -am 'Initialize Sqitch configuration.'

Add your first sql script/change:

$ sqitch add create_stg_schema -n 'Add schema for all staging objects.'

Created deploy/create_stg_schema.sql

Created revert/create_stg_schema.sql

Created verify/create_stg_schema.sql

As you can see, Sqitch creates deploy, revert a verify files for you.

$ vi deploy/create_stg_schema.sql

Add:

CREATE SCHEMA staging;

Make sure you remove the default BEGIN; COMMIT; for this as we are just creating a schema and don’t require any transaction.

$ vi revert/create_stg_schema.sql

Add:

DROP SCHEMA staging;

$ vi verify/create_stg_schema.sql

Add:

SELECT pg_catalog.has_schema_privilege('staging', 'usage');

This is quite PostgreSQL specific. For other dbs use something like this:

SELECT 1/COUNT(*) FROM information_schema.schemata WHERE schema_name = 'staging';

Now test if you can deploy the script and revert it:

Try to deploy the changes:

The general command looks like this:

$ sqitch -d <dbname> deploy

As we have already specified a default db in the config file, we only have to run the following:

$ sqitch deploy

Adding metadata tables to dwh

Deploying changes to dwh

 + create_stg_schema .. ok

Note the plus sign in the feedback which means this change was added.

When you run deploy for the very first time, Sqitch will create maintenance tables in a dedicated schema automatically for you. These tables will (among other things) store in which “version” the DB is.

Check the current deployment status of database dwh:

$ sqitch -d dwh status

# On database dwh

# Project:  yes

# Change:   bc9068f7af60eb159e2f8cc632f84d7a93c6fca5

# Name:     create_stg_schema

# Deployed: 2013-08-07 13:01:33 +0100

# By:       Diethard Steiner <diethard.steiner@>

To verify the changes run:

$ sqitch -d dwh verify

Verifying dwh

 * create_stg_schema .. ok

Verify successful

To revert the changes the the previous state, run:

$ sqitch revert --to @HEAD^ -y

Side note

You can use @HEAD^^ to revert to two changes prior the last deployed change.

To revert everything:

$ sqitch revert

Revert all changes from dwh? [Yes] Yes

 - create_stg_schema .. ok

To revert back to a specific script (you can also revert back to a specific tag):

$ sqitch revert create_dma_schema

Revert changes to create_dma_schema from dwh? [Yes]

Let’s inspect the log:

$ sqitch log

Note that the actions we took are shown in reverse chronological order, with the revert first and then the deploy.

Now let's commit it.

$ git add .

$ git commit -m 'Added staging schema.'

Now that we have successfully deployed and reverted the current change, let’s deploy again:

$ sqitch deploy

Let’s add a tag:

$ sqitch tag v1.0.0-dev1 -n 'Tag v1.0.0-dev1.'

Deployment to target DBs

So if you want to deploy these changes to your prod DB in example, you can either do it like this:

$ sqitch -d <dbname> -u <user> -h <host> -p <port> deploy

(Important: If you are working with PostgreSQL, make sure you add your password to ~/.pgpass and then comment the password out in sqitch.conf beforehand otherwise this will not work.)

Or bundle them up, copy the bundle to your prod server and deploy it there:

$ sqitch bundle

Distribute the bundle

On the prod server:

$ cd bundle

$ sqitch -d dwhprod deploy

A future version of Sqitch will have better support for target DBs (see here).

Using Sqitch with an existing project (where some ddl already exists)

Sometimes you take over a project and want to bring the existing DDL under version control and change management.

Thanks to David for providing details on this:

The easiest option is to export the existing DDL and store it in one deploy file. For the revert file you could use a statement like this then:

   DROP $schema CASCADE;

Let’s assume we call this change “firstbigchange”:

The first time you do a deploy to the existing database with Sqitch, do it twice: once with --log-only to apply your first big change, and then, from then on, without:

   $ sqitch deploy --log-only --to firstbigchange

   $ sqitch deploy --mode change

The --long-only option has Sqitch do everything in the deploy except actually run deploy scripts. It just skips it, assumes it worked successfully, and logs it. You only want to do this --to that first big dump change, as after that you of course want Sqitch to actually run deploy scripts.

Using more than one DB

DS: Currently it seems like there is a Sqitch version for each of these dbs. What if I was working on a project that used two different dbs installed on the same server and I wanted to use Sqitch for both of them (especially for dev I have more than one db installed on the same server/pc)?

DW: You can have more than one plan and accompanying files in a single project by putting them into subdirectories. They would then be effectively separate Sqitch projects in your SCM. The sqitch.conf file at the top can be shared by them all, though, which is useful for setting up separate database info for them ([core.pg] and [core.mysql] sections, for example).

If you are starting a new project, you would do it like this:

$ sqitch --engine pg --top-dir pg init myproject

$ sqitch --top-dir mysql init myproject

Then you have sqitch.plan, deploy, revert, and verify in pg/, and sqitch.plan deploy, revert, and verify in mysql/. To add a new change, add it twice:

$ sqitch --top-dir pg add foo

$ sqitch --top-dir mysql add foo

Installing Sparkl

Initial App Setup

Let’s create our first app: Tools > Sparkl. Click on the BIG plus icon, then on CREATE:

Assign a unique name and click Create Plugin:

Then you are informed that:

So once you have your normal PUC back, go to Tools> Sparkl. You should see your app now:

Click the Edit icon. Time to provide some app details:

Click on Add new Element. Add 1) a dashboard and 2) a Kettle transformation:

I am sure you can’t wait to have a look at this Kettle transformation, which has just been created for you … well, let’s go ahead and fire up Spoon and open up this file:

Creating the dashboard

Ok, let’s edit the dashboard:

1. Create a row called passwordTextRow.
2. With passwordTextRow still marked, click on the Add HTML icon.
3. Mark the HTML row and add this HTML snippet on the right hand side:
   <p>Specify your new password:</p>
4. Then add a new row called passwordInputRow. For this row, add two columns, one called passwordInputColumn and the other one passwordSubmitColumn. The layout should now look like this:
5. Save the dashboard and switch to the Component Panel.
6. Create a parameter: Generic > Simple Parameter. Call it passwordParameter.
7. Add a button: Others > Button Component. Call it passwordSubmitButton. For Label specify Submit and for HtmlObjectpasswordSubmitColumn (just press CTRL+Space to retrieve the values of the available HtmlObjects):
8. Add an Input field: Select > TextInput Component. Name it passwordTextInput, assign the ParameterpasswordParameter and the HtmlObjectpasswordInputColumn to it:
9. Now switch to the Datasource Panel. Remove the SQL dummy query.
10. From the left hand side open MYPASSWORDCHANGE Endpoints and choose mypasswordchangerendpoint Endpoint. For this datasource specify the name myPasswordChangerDS in the Properties section:
11. Switch back to the Components Panel. In the Components area select the Button Component. Click on Advanced Properties.

• For Action Parameters specify passwordParameter as [["passwordParameter"],["passwordParameter"]] and
• For Action Datasource specify myPasswordChangerDS. At the time of this writing there were considerations about moving the datasource property to main properties area (instead of advanced properties), so this might have changed by the time you read this.
• For Listeners specify passwordParameter

12. Save the dashboard.
13. Let’s see how our Sparkl plugin looks so far. Choose Tools > MyPasswordChanger:
    And you should see something like this:

Preparing the Kettle transformation

Click on UserRoleDaoResource from the REST Resources list, then choose /userroledao/updatePassword:

Let’s inspect the request body element a bit closer, so click on the user link:

1. Create a new biserver-user with admin rights which we can use just for authentication purposes.
2. Fire up Spoon and open mypasswordchangerendpoint.ktr
3. Amend the transformation to look like the one shown in the screenshot below:
4. Double click on Generate Rows. Set the Limit to 1. Create three fields:

5. Double click on Add XML. Into the Output Value field type userXML and into the Root XML element field user:
   
6. Next click on the Fields tab. Configure as shown below:
7. Double click on the REST client step. Configure as outlined below:

• General:
• URL name field: url
• HTTP method: PUT
• Body field: userXML
• Application Type: XML
• Authentication
• HTTP Login: admin-rest
• HTTP password: test123 (note: use the details of the specially created rest admin user here!)

• username: CPK server side parameter (currently logged in user)
• password: supplied by CDE dashboard
• URL: IP and port
• authentication username
• authentication password

How to pass a standard parameter from a dashboard to a Kettle job or transformation

1. Open mypasswordchangerendpoint.ktr in Spoon.
2. Right click on the canvas and choose Transformation settings.
3. Specify the a new parameter called passwordParameter. As you might have guessed, this parameter name has to be exactly the same as defined in the dashboard:
4. Change the transformation to look like this one:
   Disable the hob from the original Generate Rows step to Add XML. Add a new Generate Rows and a Get Variables step. For now the newGenerate Rows should only supply the url and user values: Right now we just want to test if the password parameter is passed on properly. Open up the Get Variables step and create a new fields called password which references the variable ${passwordParameter}:
   This setup enables us to use the passwordParameter originating from the dashboard in our Kettle transformation stream.
5. Just for testing purposes write the password field to the log (so that we see that the value is actually passed on). Change the Write to Log step config respectively.

Server-side Parameters

Kettle Properties

Sparkl server side parameters

Amending our transformation

1. Remove any fields from the new Generate rows step. It should be blank:
2. Open up the Set Variables set and configure it as shown in the screenshot … add the username variable referencing cpk.session.username:
3. Adjust the Add XML step:
4. Change the config of the REST Client step to not read any more the URL from the field but instead key it into the URL config field like this:
   ${VAR_PENTAHO_BISERVER_URL}/api/userroledao/updatePassword
5. Also, in the Authentication reference the parameters we just set up: ${VAR_PENTAHO_BISERVER_USER} and ${VAR_PENTAHO_BISERVER_PW} respectively.
6. Save and then restart the server.
7. Then test again the dashboard and watch the Tomcat log.

Closing words

Why Bootstrap

How to use it with CDE

Configure your CDE Dashboard to use Bootstrap

The very first thing to do is define the settings for your dashboard. In CDE click on the Settings button in the main menu:

In the settings screen provide a Name and then set the Dashboard Type to Bootstrap:

Generate your standard layout

• With Bootstrap, the total span size of the page is 12 columns (as opposed to Blueprint, which has 24 columns). This means that if your page has only one column, the span size should be 12. If your page has 2 columns, the span size should be 6 for each of them and so on (calc: 12 / no of cde columns).
• Within each column nest an HTML element: This one will hold the Bootstrap HTML snippet. In the most simple form it will look like this:
  Make sure to provide a name for the HTML object in the Properties panel (otherwise, the styles do not show up properly in the preview - at least in my case this happened).

Get code from Bootstrap website

The Bootstrapwebsite offers a wealth of components. These ones can be found by click in the main menu on either CSS or Components. Use the side panel menu to get detailed info about each component:

One component which you will most likely use is called Penals [reference]. Again, there is a wide choice of panels available, let’s take a look at the Penal with header:

Amend Bootstrap HTML

1. Open up JSFiddle in a web browser tab.
2. Copy the Bootstrap HTML snippet into the HTML panel
3. Copy this reference into the CSS panel: @import url('http://getbootstrap.com/dist/css/bootstrap.css')
4. Adjust the HTML. Do at least the following:
1. Provide a proper title.
2. Add an id attribute to the content div. This way we can later on reference it when we want to assign a chart component.
3. Delete the default content text.
5. Then click the Run button to get a preview:

Add amended Bootstrap HTML to CDE Layout Structure

When we initially created the Layout Structure in CDE, we nested an HTML element inside each column - just copy bootstrap HTML there. Choose the HTML element in the Layout Structure panel, then click on the HTML ellipsis [...] button in the Properties panel:

Finally paste the Bootstrap HTML snippet:

Save the dashboard now and do a preview:

Create your data sources

Create your components

So for this article I quickly created a CCCBarChart component. The important bit about it is that I specified chart1for the HTMLObject property:

Note that this ties back to the div id attribute we defined in our Bootstrap HTML snippet [stored in the Layout Structure HTML element]:

So let’s do another preview:

Last Example

1. Adjust the Layout Structure of your dashboard to make room for a button.
2. Visit the Bootstrap website to find out all about their fancy buttons: reference.
3. Let’s copy the HTML snippet for the standard button. Jump over to JSFiddle and adjust it until you are happy with the preview. Remember to add a dedicated id attribute.
4. Copy the HTML snippet again and paste it into the HTML properties field of your HTML element in the Layout Structure panel.
5. Save and Preview your dashboard.
   Again, a very easy example. I guess now you are certainly interested in creating some more challenging dashboards with Pentaho CDE and Bootstrap!

Pentaho Dashboards CDE: Create your custom table

You want to implement something in your dashboard that is not covered by the out-of-the-box dashboard components? Luckily, with Pentaho CDE the world is open: CDE makes use of the standard web technologies (CSS, JavaScript, HTML), so theoretically you can implement whatever is in the realm of these technologies. Obviously you will need some basic knowledge of these technologies (setup is not as easy any more as filling out some config dialogs), but the possibilities are endless. In this post I’ll briefly talk you through how to source some data and then to create a custom table with it (which you can easily do with one of the CDE components as well, but that’s not the point here … imagine what else you could do):

1. In CDE, register a Datasource. In example create a sql over sqlJndi datasource, provide a Name i.e. qry_generic_select, choose SampleData for JNDI and specify following query:

    SELECT customername, customernumber, phone FROM customers
   

2. In the component section, add a Query Component. This component is most commonly used for displaying simple results, like one number in a dashboard (i.e. max temperature). Here we will use this component to retrieve a bigger result set.
3. Click on Advanced Properties.
4. For the Datasource property specify the datasource you created in step 1 (i.e. qry_generic_select)
5. Provide a name for the Result Var. This is the variable, which will hold the output data of your datasource.

6. Write a Post Execution function, in example:

    function() {
         document.getElementById('test').innerHTML = JSON.stringify(select_result);
    } 
   

7. We will only use this function for now to test if the query is working. Later on we will change it.

8. The setup so far should look like this:
9. In the Layout Panel create a basic structure which should at least have one column. Name the column test as we referenced it already in our JavaScript function.
10. Preview your dashboard (partial screenshot):

11. Let’s change the Post Execution function to return only the first record:

    function() {
         document.getElementById('test').innerHTML = JSON.stringify(select_result[0]);
    }
    

    And the preview looks like this:

12. Let’s change the Post Execution function to return only the first entry from the first record:

    function() {
         document.getElementById('test').innerHTML = JSON.stringify(select_result[0][0]);
    }
    

    And the preview looks like this:

13. Let’s extend our Post Execution function to create a basic table:

    function() {
      var myContainer = document.getElementById('test');
      var myTable = document.createElement('table');
      var myTr = document.createElement('tr');
      var myTd = document.createElement('td');
    myContainer.appendChild(myTable).appendChild(myTr).appendChild(myTd).innerHTML = select_result[0][0];
    }
    

    Do a preview and make use of your browser’s developer tools to see the generated HTML:

14. Ok, now that this is working, let’s add some very basic design. Click on Settings in the main CDE menu:

15. Choose bootstrap from the Dashboard Type pull down menu: Click Save.

16. Back to the Post Execution function of the Query Component: Now we want to make this a bit more dynamic: For every data row must be enclosed by <td> and within each data row each data value must be enclosed by <td>. We also have to add the <tbody> element to make a proper table. And we will apply the Bootstrap Striped Table design:

    // Simple function preparing the table body 
    
    function() {
        var myContainer = document.getElementById('test');
        var myTable = document.createElement('table');
        var myTBody = document.createElement('tbody');
        var myTr = document.createElement('tr');
        var myTd = document.createElement('td');
    
        //myTable.id = 'table1';
        myTable.className = 'table table-striped';
    
        myContainer.appendChild(myTable).appendChild(myTBody);
    
        for(var i = 0; i < select_result.length; i++) {
            myContainer.lastChild.lastChild.appendChild(myTr.cloneNode());
            for(var j = 0; j < select_result[i].length; j++) {
                myText = document.createTextNode(select_result[i][j]);
                myContainer.lastChild.lastChild.lastChild.appendChild(myTd.cloneNode()).appendChild(myText);
            }
        }
    }
    

    You can find a text version of this JavaScript code a bit further down as well in case you want to copy it.

17. Do a preview now and you will see that we have a basic table now:

    Note: In case you are creating this dashboard as part of a Sparkl plugin and you are having troubles seeing the bootstrap styles applied (and are sure that the problem is not within your code), try to preview the dashboard from within your Sparkl project endpoint listing (which seems to work better for some unknown reason):

18. One important thing missing is the header. Let’s source this info now. The Query Component provides following useful functions, which you can access within Post Execution function:

    this.metadata
    this.queryInfo
    this.resultset
    

    To get an idea of what is exactly available with in the metadata object, you can use in example this function:

      document.getElementById('test').innerHTML = JSON.stringify(this.metadata);
    

    Which reveals the following:

19. This is the function preparing the full table (header and body):

    // function preparing the full table (header and body)
    
    function() {
    
        var myContainer = document.getElementById('test');
        var myTable = document.createElement('table');
        var myTHead = document.createElement('thead');
        var myTh = document.createElement('th');
        var myTBody = document.createElement('tbody');
        var myTr = document.createElement('tr');
        var myTd = document.createElement('td');
    
        //myTable.id = 'table1';
        myTable.className = 'table table-striped';
    
        //document.getElementById('test').innerHTML = JSON.stringify(this.metadata);
        myMetadata = this.metadata;
    
        myContainer.appendChild(myTable).appendChild(myTHead).appendChild(myTr);
    
        for(var s = 0; s < myMetadata.length; s++){
            myHeaderText = document.createTextNode(myMetadata[s]['colName']);
            myContainer.lastChild.lastChild.lastChild.appendChild(myTh.cloneNode()).appendChild(myHeaderText);
        }
    
        myContainer.lastChild.appendChild(myTBody);
    
        for(var i = 0; i < select_result.length; i++) {
            myContainer.lastChild.lastChild.appendChild(myTr.cloneNode());
            for(var j = 0; j < select_result[i].length; j++) {
                myText = document.createTextNode(select_result[i][j]);
                myContainer.lastChild.lastChild.lastChild.appendChild(myTd.cloneNode()).appendChild(myText);
            }
        }
    
    }
    

20. And the preview looks like this:

Voilá, our custom boostrap table is finished. This is not to say that you have to create a table this way in CDE: This was just an exercise to demonstrate a bit of the huge amount of flexibility that CDE offers. Take this as a starting point for something even better.

Setting a variable value dynamically in a Pentaho Data Integration job

On some occasions you might have to set a variable value dynamically in a job so that you can pass it on to the Execute SQL Script job entry in example. In this blog post we will take a look at how to create an integer representation of the date of 30 days ago. And we want to achieve this without using an additional transformation!

The way to achieve this in a simple fashion on the job level is to use the Evaluate JavaScript job entry [Pentaho Wiki]. While this job entry is not really intended to do this, it currently offers the easiest way to accomplish just this. Just add this job entry to your Kettle job and paste the following JavaScript:

date = new java.util.Date();
date.setDate(date.getDate()-30); //Go back 30 full days
var date_tk_30_days_ago = new java.text.SimpleDateFormat("yyyyMMdd").format(date);
parent_job.setVariable("VAR_DATE_TK_30_DAYS_AGO", date_tk_30_days_ago);
true; // remember that this job entry has to return true or false

To test this let's add a Log job entry:

Add this to the log message to the job entry settings:

The date 30 days ago was: ${VAR_DATE_TK_30_DAYS_AGO}

And then run the job. You should see something similar to this:

Certainly you could just pass the value as parameter from the command line to the job, but on some occasions it is more convenient to create the value dynamically inside the job.

Software used:

• pdi-4.4.0-stable

1. If it is possible to access the name field (in our case CITY) via the SQL query, we could change our main report SQL query and add it there as a new field. But this is not very efficient, right?
2. We could create a new query which takes the code/id (in our case OFFICECODE) as a parameter and returns the name (CITY) and then run this query in a sub-report which could return the value to the main report (this is in fact what you had to do some years back). Well, not that neat either.
3. Here comes the savior: The SINGLEVALUEQUERY formula function. You can find this one in the Open Formula section. Thomas posted some interesting details about it on his blog some time ago.

1. We create an additional query called ds_office_chosen which is constrained by the code/id and returns the (display) name: SELECT city AS office_chosen FROM offices WHERE officecode = ${param_officecode}
2. We create a new formula element called formula_office_chosen and reference the query ds_office_chosen: =SINGLEVALUEQUERY("ds_office_chosen")
3. We can now use formula_office_chosen in our report:

How to work with MapReduce Key Value Pairs in Pentaho Data Integration

Pentaho Kettle Parameters and Variables: Tips and Tricks

Parameters and Variables

Definitions upfront

Example

jb_main.kjb

jb_slave.kjb

tr_set_variables.ktr

tr_show_param_values.ktr

Making Parameters available for all subprocesses in an easy fashion

1. In the main job specify the parameters that you want to pass in in the Job Settings:
   This way parameters and their values can be passed in from the command line in example.
2. Right after the Start job entry use the Set Variables job entry. Specify the variable names, reference the parameters you set up in step 1 and set the scope to Valid in the current job.
3. There is no need to specify any parameters/variables in any of the subprocesses.

What if I still want to be able to run my subprocess independently sometimes?

1. In the subprocess, specify the parameter that you want to be able to pass in. In our example (which is based on the previous example), we modified the transformation tr_show_param_values.ktr and added following parameters to the Transformation Settings:
   We also amended the Get Variables step to make use of these parameters:
   This way, we can already run this transformation on its own. Now we only have to adjust the parent job so that we can run it from there as well.
2. In the parent job, in the Job or Transformation job entry settings, go to the Parameters tab and tick Pass all parameter values down to the sub-transformation/sub-job. Next, as the Parameter set the name of the parameter you defined in the subprocess. As the Value define the variable that you want to pass down: ${variable}. This assumes that this variable was set beforehand by some Set Variables job entry/step.
   In our case, we modified transformation job entry in the job jb_slave.kjb and added following mapping to the job entry settings in the Parameters tab:

Closing remarks

Subtitle: Kettle Testing for the Non-Java Developers

Announcing the KTF Beta:

Precautions

The small print upfront

Motivation

Contribute!

Features

• Supports multiple input datasets
• Supports multiple output datasets
• Supports sorting of the output dataset so that a good comparison to the golden output dataset can be made
• The setup is highly configurable (but a certain structure is enforced - outlined below)
• Non conflicting parameter/variable names (all prefixed with “VAR_KTF_”)
• Non intrusive: Just wraps around your existing process files (except some parameters for db connections etc will have to be defined … but probably you have this done already anyways)

Current shortcomings

• Not fully tested (only tested with included samples and on PostgreSQL)
• Currently works only with Table input / output transformations. Text/CSV file input/output will be supported in a future version (which should not be too complicated to add).
• Dictates quite a strict folder structure
• Limited documentation

Project Folder Structure

1. Adjust parameter values in config/.kettle/kettle.properties and JNDI details in config/simple-jndi/jdbc.properties
2. Your transformations have to be prefixed for now with “tr_” and your jobs with “jb_”. Do not use any special characters or spaces in your job/transformation names. Your transformations and jobs have to be saved within repository/main. There is currently now subfolder structure allowed within this folder.
3. Within the config/test-cases directory create a folder for the processes you want to test. A process can be a transformation or a job. Name these folders exactly the same as the job/transformation you want to test (just without the file extension).  Each process folder must have an input and output folder which hold the DDL, Kettle data type definitions and in case of the output the sort order definition (see tr_category_sales sample on how to set this up). If your output dataset does not require any sorting, create an empty sort def file (see tr_fact_sales example). Note that KTF can handle more than one output/input dataset.
4. The process folder must also contain at least one test case folder (which has to have a descriptive name). In the screenshot above it is called “simpletest”. A test case folder must contain an input and output folder which each hold the dedicated datasets for this particular test case. In case of the output folder it will hold the golden output dataset(s) (so that dataset that you want to compare your ETL output results to).
5. Users working on Windows: For all the CSV output steps in transformations under /repository/test set the Format to Windows (Content tab). KTF has not been tested at all on Windows, so you might have to make some other adjustments as well.
6. Add environment variables defined in config/set-project-variables.sh to your .bashrc. Then run: source ~/.bashrc
7. Start Spoon (this setup requires PDI V5) or run the process from the command line.
8. Run the test suite
9. Analyze results in tmp folder. If there is an error file for a particular test case, you can easily visually inspect the differences like this:

Future improvements

• Write test results to dedicated database table
• Improvement of folder structure
• Support for text file input and output for main processes (jobs/transformations)

FAQ

Going Agile: Sqitch Database Change Management

You have your database scripts under a dedicated version control and change management system, right? If not, I recommend doing this now.

While there have been handful of open source projects around which focus on DB script versioning and change management control, none has really gained a big momentum and a lot of them are dormant.

But there is a new player on the ground! A light at the end of the db change management tunnel - so to speak. David Wheeler has been working on Sqitch over the last year and the results are very promising indeed! Currently the github projects shows 7 other contributors, so let’s hope this project gets a strong momentum! Also a new github project for a Sqitch GUI was just founded.

Why I like Sqitch:

• You can run all the commands from the command line and get very good feedback.
• Everything seems quite logical and straightforward: It’s easy to get to know the few main commands  and in a very short amount of time you are familiar with the tool.
• You can use your choice of VCS.
• It works very well.

Supported DBs are currently MySQL, Oracle, SQLite and PostgreSQL. CUBRID support is under way.

So what do we want to achieve? So what do we want to achieve?

Bring all DDL, stored procedures etc under version control. This is what Git is very good for (or your choice of CVS).

Keep track of the (order of) changes we applied to the database, verify that they are valid, be able to revert them back to a specific state if required. Furthermore, we want to deploy these changes (up to a specific state) to our test and production databases. This is was Sqitch is intended for:

The below write-up are my notes partially mixed with David’s ones.

Info

http://sqitch.org/

https://github.com/theory/sqitch

https://metacpan.org/module/sqitchtutorial

Forum:

https://groups.google.com/forum/#!forum/sqitch-users

Installation

Options:

PostgreSQL: cpan App::Sqitch DBD::Pg (You also have to have PostgreSQL server installed)

SQLite: cpan App::Sqitch DBD::SQLite

Oracle: cpan App::Sqitch DBD::Oracle (You also have to have SQL*Plus installed)

MySQL: cpan App::Sqitch

If you want to have support for i.e. PostgreSQL and Oracle you can just run:

PostgreSQL: cpan App::Sqitch DBD::Pg DBD::Oracle

For more install options see here.

Below I will only discuss the setup for PostgreSQL.

# make sure you have build tools install (make etc) sudo apt-get install built-essential

# sqitch requires Locals support (asks for Locale::TextDomain and Locale::Messages)
sudo apt-get install libintl-perl
# make sure pg_config is installed sudo apt-get install libpq-dev # RHEL (Fedora, CentOs, etc) use instead: # sudo yum install postgresql-devel # Note where the psql include dir is located. You might be asked for it on installing sqitch pg_config --includedir # Note the version of psql. You might be asked for it on installing sqitch psql --version # install sqitch (adjust for your DB) sudo cpan App::Sqitch DBD::Pg 

Once installation is finished, check out the man page:

$ man sqitch

Within your git project directory, create a dedicated folder:

$ mkdir sqitch

$ git add .

$ cd sqitch

$ sqitch --engine pg init projectname

Let's have a look at sqitch.conf:

$ cat sqitch.conf

Now let’s add the connection details:

$ vi sqitch.conf

uncomment and specify:

[core "pg"]

client = psql

username = postgres

password = postgres

db_name = dwh

host = localhost

port = 5432

# sqitch_schema = sqitch

If psql is not in the path, run:

$ sqitch config --user core.pg.client /opt/local/pgsql/bin/psql

Add your details:

$ sqitch config --user user.name 'Diethard Steiner'

$ sqitch config --user user.email 'diethard.steiner@'

Let’s add some more config options: Define the default db so that we don’t have to type it all the time:

$ sqitch config core.pg.db_name dwh

Let's also make sure that changes are verified after deploying them:

$ sqitch config --bool deploy.verify true

$ sqitch config --bool rebase.verify true

Check details:

cat ~/.sqitch/sqitch.conf

Have a look at the plan file. The plan file defines the execution order of the changes:

$ cat sqitch.plan

$ git add .

$ git commit -am 'Initialize Sqitch configuration.'

Add your first sql script/change:

$ sqitch add create_stg_schema -n 'Add schema for all staging objects.'

Created deploy/create_stg_schema.sql

Created revert/create_stg_schema.sql

Created verify/create_stg_schema.sql

As you can see, Sqitch creates deploy, revert a verify files for you.

$ vi deploy/create_stg_schema.sql

Add:

CREATE SCHEMA staging;

Make sure you remove the default BEGIN; COMMIT; for this as we are just creating a schema and don’t require any transaction.

$ vi revert/create_stg_schema.sql

Add:

DROP SCHEMA staging;

$ vi verify/create_stg_schema.sql

Add:

SELECT pg_catalog.has_schema_privilege('staging', 'usage');

This is quite PostgreSQL specific. For other dbs use something like this:

SELECT 1/COUNT(*) FROM information_schema.schemata WHERE schema_name = 'staging';

Now test if you can deploy the script and revert it:

Try to deploy the changes:

The general command looks like this:

$ sqitch -d <dbname> deploy

As we have already specified a default db in the config file, we only have to run the following:

$ sqitch deploy

Adding metadata tables to dwh

Deploying changes to dwh

 + create_stg_schema .. ok

Note the plus sign in the feedback which means this change was added.

When you run deploy for the very first time, Sqitch will create maintenance tables in a dedicated schema automatically for you. These tables will (among other things) store in which “version” the DB is.

Check the current deployment status of database dwh:

$ sqitch -d dwh status

# On database dwh

# Project:  yes

# Change:   bc9068f7af60eb159e2f8cc632f84d7a93c6fca5

# Name:     create_stg_schema

# Deployed: 2013-08-07 13:01:33 +0100

# By:       Diethard Steiner <diethard.steiner@>

To verify the changes run:

$ sqitch -d dwh verify

Verifying dwh

 * create_stg_schema .. ok

Verify successful

To revert the changes the the previous state, run:

$ sqitch revert --to @HEAD^ -y

Side note

You can use @HEAD^^ to revert to two changes prior the last deployed change.

To revert everything:

$ sqitch revert

Revert all changes from dwh? [Yes] Yes

 - create_stg_schema .. ok

To revert back to a specific script (you can also revert back to a specific tag):

$ sqitch revert create_dma_schema

Revert changes to create_dma_schema from dwh? [Yes]

Let’s inspect the log:

$ sqitch log

Note that the actions we took are shown in reverse chronological order, with the revert first and then the deploy.

Now let's commit it.

$ git add .

$ git commit -m 'Added staging schema.'

Now that we have successfully deployed and reverted the current change, let’s deploy again:

$ sqitch deploy

Let’s add a tag:

$ sqitch tag v1.0.0-dev1 -n 'Tag v1.0.0-dev1.'

Deployment to target DBs

So if you want to deploy these changes to your prod DB in example, you can either do it like this:

$ sqitch -d <dbname> -u <user> -h <host> -p <port> deploy

(Important: If you are working with PostgreSQL, make sure you add your password to ~/.pgpass and then comment the password out in sqitch.conf beforehand otherwise this will not work.)

Or bundle them up, copy the bundle to your prod server and deploy it there:

$ sqitch bundle

Distribute the bundle

On the prod server:

$ cd bundle

$ sqitch -d dwhprod deploy

A future version of Sqitch will have better support for target DBs (see here).

Using Sqitch with an existing project (where some ddl already exists)

Sometimes you take over a project and want to bring the existing DDL under version control and change management.

Thanks to David for providing details on this:

The easiest option is to export the existing DDL and store it in one deploy file. For the revert file you could use a statement like this then:

   DROP $schema CASCADE;

Let’s assume we call this change “firstbigchange”:

The first time you do a deploy to the existing database with Sqitch, do it twice: once with --log-only to apply your first big change, and then, from then on, without:

   $ sqitch deploy --log-only --to firstbigchange

   $ sqitch deploy --mode change

The --long-only option has Sqitch do everything in the deploy except actually run deploy scripts. It just skips it, assumes it worked successfully, and logs it. You only want to do this --to that first big dump change, as after that you of course want Sqitch to actually run deploy scripts.

Using more than one DB

DS: Currently it seems like there is a Sqitch version for each of these dbs. What if I was working on a project that used two different dbs installed on the same server and I wanted to use Sqitch for both of them (especially for dev I have more than one db installed on the same server/pc)?

DW: You can have more than one plan and accompanying files in a single project by putting them into subdirectories. They would then be effectively separate Sqitch projects in your SCM. The sqitch.conf file at the top can be shared by them all, though, which is useful for setting up separate database info for them ([core.pg] and [core.mysql] sections, for example).

If you are starting a new project, you would do it like this:

$ sqitch --engine pg --top-dir pg init myproject

$ sqitch --top-dir mysql init myproject

Then you have sqitch.plan, deploy, revert, and verify in pg/, and sqitch.plan deploy, revert, and verify in mysql/. To add a new change, add it twice:

$ sqitch --top-dir pg add foo

$ sqitch --top-dir mysql add foo