Overview ⚙️

Note

Adapted from Hashicorp's gRPC KV Plugin example and RPC-based plugins in Go by Eli Bendersky (@eliben) 🚀

This repo contains Funnel Plugin development using the go-plugin package from HashiCorp.

In this setup, the Plugin handles all user authentication, with the Server having no knowledge or record of specific user credentials/tokens.

Quick Start ⚡

1. Build the Plugin and Test Server

First build and run the test User Database server:

➜ git clone https://github.com/ohsu-comp-bio/funnel-plugins

➜ cd funnel-plugins

➜ make

Tip

This will create three executable binaries in the build directory:

Executable	Description
cli	the "application" used for invoking the plugin binary
test-server	the Test Server used for storing user and credentials (called by the plugin)
plugins/authorizer	the plugin binary

2. Start the Test Server

➜ ./test-server
Server is running on http://localhost:8080

3. Test Authorized User

Then in another terminal window, invoke the CLI component to authenticate a user named example who is an Authorized user (i.e. found in the "User Database" — example-users.csv):

➜ ./build/cli example
{
  "code": 200,
  "message": "User found!",
  "config": {
    "Key": "key1",
    "Secret": "secret1"
  }
}

4. Test Unauthorized User

Attempt to authenticate a user named error, representing an Unauthorized user:

➜ ./build/cli error
{
  "code": 401,
  "message": "User not found"
}

Architecture 📐

This repo contains the following major components:

1. Plugin — the actual plugin itself, makes calls to the external User Database Service
2. CLI — the helper program that allow users to run the plugin from the command line
3. Test Server — the mock User Database Service to store the users and their tokens/credentials

Sequence Diagram 📝

sequenceDiagram
    title Funnel Plugin + Per-User S3 Bucket Interactions

    participant User
    participant FunnelServer as Funnel Server
    participant FunnelWorker as Funnel Worker
    participant AuthPlugin as Auth Plugin
    participant UserDB as User Database
    participant S3Bucket as S3 Bucket

    User->>FunnelServer: Request:<br/>1) Auth Header<br/>2) TES Task<br/>3) Name of Plugin to use
    FunnelServer->>AuthPlugin: Request:<br/>1) Auth Header<br/>2) TES Task
    AuthPlugin->>UserDB: Username from Auth Header
    UserDB->>AuthPlugin: S3 Credentials
    AuthPlugin->>FunnelWorker: S3 Credentials
    FunnelWorker->>S3Bucket: Signed Request
    S3Bucket->>User: S3 Object

Loading

Authoring Plugins ✍️

Overview 🌀

The following includes examples and resources for writing Plugins (in Go, Python, or any other supported language!)  

• gRPC Example (this is largely what Funnel Plugins is based off of, along with this manager snippet by Eli Bendersky for loading the plugin binaries)  
• Intro (super helpful reference from beginning to end)

Communicating with Funnel

Warning

TODO: Add the following to the docs 🚧

• API "contract" between the Plugin and Funnel Server":
  • What exactly will the Plugin require for inputs and outputs (Config)?
  • What functions will plugin authors need to implement (e.g. Get)?

Inputs

User string

Outputs

config.Config

Functions

Authorize.Get()

Python 🐍

For authoring custom plugins in Python, see the example Python plugin!

gRPC + Protobuf ️🚀

Tip

Understanding gRPC and protobufs isn't necessary to writing plugins, but it can be helpful when errors or bugs arise 🐛

Under the hood, all communication between the Plugin and the Funnel Server happens over gRPC using Protocal Buffers (protobufs).

• Protobuf Overview   
• Tutorials for Go and Python  
• Awesome gRPC — pretty up-to-date resource for all things Protobuf and gRPC! 😎

Additional Resources 📚

• https://github.com/hashicorp/go-plugin
• https://pkg.go.dev/github.com/hashicorp/go-plugin
• https://eli.thegreenplace.net/2023/rpc-based-plugins-in-go
• https://github.com/eliben/code-for-blog/tree/main/2023/go-plugin-htmlize-rpc