Injecting Custom viz Plugins in Apache Superset

Motivation: This article is detailed documentation I made while following Nielsen’s articleon injecting plugins & injecting my own plugins into Superset.

While following Nielsen’s articleon injecting plugins, I ran into a bunch of errors & failed approaches to fix them. Hence, I decided to document the approach step-by-step which finally worked for me.

The process is divided into 3 parts:
1. Setting up Superset
2. Injecting the custom viz plugin
3. Testing the image

Setting up Superset

1. Clone the apache superset repo:
   git clone https://github.com/apache/superset
cd into the home folder of the repo & switch to the version of superset you want to inject your image in. Here we are switching to branch 1.3: git checkout 1.3.

2. Pull the docker image of the Superset version 1.3(or the version you are using) published by Nielsenoss on DockerHub:

docker pull nielsenoss/apache-superset:1.3

3. Run the image in interactive mode: docker run -it nielsenoss/apache-superset:1.3 /bin/sh& copy the superset-frontend folder from the container to your host system using:

docker cp <containerId>:/app/superset-frontend ./

4. Replace the folder superset/superset-frontend with the superset-frontend folder you extracted from the Nielsenoss container.

Replacing the superset-frontend folder will remove errors in the .js & .ts files you would encounter when trying to build the Superset image with injected plugins. A lot of changes need to be made in the various components of the frontend to remove these errors, which were fixed in the Nielsenoss image. To see these differences, run the diff command on the original superset-frontend folder & the one from Nielsenoss:

diff -qr superset-frontend nielsenoss-superset-frontend

where nielsenoss-superset-frontend is the superset-frontend folder from the nielsenoss docker image.

5. Make sure the version of node & npm installed on your machine match the ones specified in package.json

Injecting your custom plugin

Source: https://medium.com/nmc-techblog/apache-superset-manage-custom-viz-plugins-in-production-9fde1a708e55

1. Clone nielsen-oss superset-viz-plugins repo in the same directory as that of superset:

git clone https://github.com/nielsen-oss/superset-viz-plugins

2. Make sure your plugin is published on npm registry. Here we take the npmjs organisation name as improved-octo-succotash & the plugin name as plugin-chart-hello-world as examples for the next steps.

3. Copy the code of your plugin to superset-viz-plugins/plugins

4. cd into the directory superset/superset-frontend

5. Add your npm organisation to the .npmrc file. In this example, add the line:

@improved-octo-succotash:registry=http://registry.npmjs.org/

6. Run the below command with changes made according to the naming used by you:

GITHUB_WORKSPACE=/Users/atirek/code PROJECT_WORKING_DIRECTORY=superset-viz-plugins node ../../superset-viz-plugins/scripts/addDependencies.js plugin-chart-hello-world

• GITHUB_WORKSPACE: The path of the directory containing the superset & superset-viz-plugins
• PROJECT_WORKING_DIRECTORY: The repo to which you copied the plugin code. To be set as ‘superset-viz-plugins’.
• plugin-chart-hello-world: The name of the folder containing your plugin code. It is passed as an argument to the .js script.
  What this step does is injects your published plugin & its version into the package.json

7. Next run this command with changes made according to the naming used by you:

GITHUB_WORKSPACE=/Users/atirek/code PROJECT_WORKING_DIRECTORY=superset-viz-plugins PRESET_NAME=HelloWorldChartPlugin node ../../superset-viz-plugins/scripts/generatePreset.js plugin-chart-hello-world

• PRESET_NAME: Name of the preset file generated by above command.
  The above command creates a preset for your plugin. Now change the preset extension to .js & move it to the presets folder:

mv HelloWorldChartPluginPreset.ts HelloWorldChartPluginPreset.js  
mv HelloWorldChartPluginPreset.js ./src/visualizations/presets/

8. Next run this command with changes made according to the naming used by you:

GITHUB_WORKSPACE=/Users/atirek/code PROJECT_WORKING_DIRECTORY=superset-viz-plugins PRESET_NAME=HelloWorldChartPlugin PLUGINS_EXTRA_FILENAME=HelloWorldPluginsExtra node ../../superset-viz-pluginsh/scripts/generateSetupPluginsExtra.js plugin-chart-hello-world

• PLUGINS_EXTRA_FILENAME: The .ts plugins file generated for your plugin.
  Change the extension of the plugins file generated to .ts & copy it to the setup folder:

mv HelloWorldPluginsExtra HelloWorldPluginsExtra.ts  
cp HelloWorldPluginsExtra.ts ./src/setup/

Also change the function name in HelloWorldPluginsExtra.ts to HelloWorldPluginsExtra() (same as the file name).

9. Call the function from the HelloWorldPluginsExtra.ts file you generated to the setupPlugins.ts file. First import the HelloWorldPluginsExtra.ts file by adding the line:

import HelloWorldPluginsExtra from './HelloWorldPluginsExtra';

Next, in the setupPlugins() function, call the function from HelloWorldPluginsExtra.ts: HelloWorldPluginsExtra();
Now your plugin has been added to Superset. To use it we rebuild the docker image.

10. One last step before building the image is to update the package-lock.json with the command: npm install --package-lock-only --legacy-peer-deps

Building & Running Superset image with plugin

Now that you plugin has been added to Superset, to see it running, rebuild the Superset docker image from the home folder of the superset repo:

docker build -t "supersetwithplugin" .

Make sure sufficient RAM(atleast 3 GB) is assigned to docker & it passes the memory check displayed 5–10 seconds after running the command.
After the image has been built replace the default superset image with the one you created in the docker-compose-non-dev.yml by replacing *superset-image under the image entry by supersetwithplugin (or the image name you assigned).
Now run the superset-services & you will be able to see your plugin:

docker-compose -f docker-compose-non-dev.yml build
docker-compose -f docker-compose-non-dev.yml up