Shuffle Architecture

The core services and their network boundaries:

[!TIP] Manual Service Health Verification via CLI
You can verify the health of your local or clustered deployment using standard commands:

API Call

GET

cURL

Python

HTTP

CopyTest

curl -X GET 'http://localhost:5001/api/v1/health'

Response

Close

No response received.

Runtime Orchestration

How do workers and apps actually get deployed? We support four primary deployment architectures:

1. Single Server / Docker Compose: Default deployment for labs, small teams, and quick setups. Server and Orborus live on the same physical host or virtual machine sharing /var/run/docker.sock.
2. Distributed Docker Swarm: Orborus schedules worker and app containers across a multi-node Swarm overlay network (shuffle_swarm_executions).
3. Cloud Hybrid (Shuffle Cloud + Local Orborus): Run your control plane in Shuffle Cloud while deploying Orborus inside your private VPC or on-prem data center with zero inbound firewall ports required.
4. Kubernetes Cloud-Native: Production-scale deployment using Helm charts. Orborus and workers leverage Kubernetes API tokens inside the shuffle namespace to provision isolated pods and services dynamically.

Frameworks

Shuffle uses and is built upon existing, well established frameworks to help the Security community move forward, rather than just increase complexity.

• OpenAPI - Used to load and generate API specifications that's applicable for other platforms than Shuffle. This is a standard that's widely used by almost every enterprise, but the security industry are lacking behind.
• Mitre Att&ck - Used to create general purpose workflows to make onboarding and usage as smooth as possible. Mitre Att&ck detections are directly proportional to risk, and can be used for KPI's to sell reasons for investment in the platform to leadership etc.
• CACAO - A standardized way of handling collaborative automation. Shuffle is built for the masses, and we intend to be as close to the CACAO specification as possible long-term. This is a work in progress (October 2022).
• Cytoscape - A light-weight and scalable design system, used for node relationships in our Workflow view. Cytoscape provides us all the necessary parts to create a fully functional workflows for any use.

Automation Engine

The Shuffle automation engine is entirely built from scratch, relying heavily on Docker and Micro-service, real-time executions. The code for decision making of next nodes can be found here, while the SDK for how Apps perform can be found here. Read more here

Technologies

The fundamental building blocks of Shuffle are all designed to be modular Docker images, meaning they can run separately in different environments. The list below contains all the necessary parts to execute a workflow. In case you want to contribute, we've added the programming languages as well.

The reason behind the usage of Golang is simple: Stability. Scripting languages like Python are prone to crashing, while Golang is fun, stable and easy to understand.

Versioning

Shuffle uses semantic versioning. All Docker images can be found on Github or on Dockerhub.

How it works

Shuffle is a quite complex platform, with lots of different features to handle anything necessary for automation. We are an API-first platform, meaning we always build the API for the feature, before developing the frontend. This allows us to quickly prototype and release new features, without necessarily breaking it. Most of our API endpoints use the following model to authentication and authorize whether the user has access:

1. User clicks a button in the frontend

2. Backend receives a HTTP request with a Session Token

3. The backend validates it the request in this order:

   1. CORS
   2. Authenticate: Are they logged in?
   3. Authorize: Do they have access to the resource?

4. If valid, the backend processes the request by accessing info in this order:

   1. Is there cache for the resource?
   2. Does the resource exist in the database? Request is authenticated in the database, before a return.

5. The data gathered is processed, before Shuffle returns either of these codes (with certain exceptions):

   • 200: Everything worked!
   • 301: Redirect to a different page to fix the issue.
   • 401: Authentication and Authorization issues.
   • 500: Internal error. Please contact us if this occurs.

   If there's a failure, Shuffle should ALWAYS return with data in this format:

   code

   {"success": False, "reason": "Here's the reason it didn't work"}
   

6. The frontend takes the data and shows it in the UI.

Authentication

There are two types of authentication tokens in Shuffle: API/Session access, and App authentication.

Data access model

Accessing data is done PER TENANT, based on which one your user is in. This means that you need to be a part of the tenant you're using the API for, and it will by default use this one unless you use tenant-Id header in your request. If you are an tenant admin, you have access to all information within an tenant, while tenant-Users has access to read and modify Workflows, Apps, Files, Datastore and Trigger management. If you are a tenant-Reader, you can READ the same data the tenant User can read and modify.

Encryption and hashing

Hashed (bcrypt):

• User passwords.

Encrypted (AES-256):

• App authentication, Protected Datastore Keys and Files are being encrypted. The seed used for hashing is random for each tenant, and can be set with the environment variable SHUFFLE_ENCRYPTION_MODIFIER in the local version of Shuffle. This is automatically handled in our SaaS offering. How it works:

  1. Create md5 hash from tenant ID + Workflow_id + Auth timestamp + SHUFFLE_ENCRYPTION_MODIFIER
  2. Encrypt the authentication value with aes.NewCipher
  3. Base64 encode the encrypted value (because bytes and strings aren't friends)
  4. Store the value in the database.

	Run in reverse to decrypt and retrieve the values.

Encryption Code reference

• Encrypt
• Decrypt

Password Management

On the cloud instance of Shuffle (https://shuffler.io) the following password policy applies:

• Minimum length of 10
• Lower- and Uppercase character

If you try to log in too many times in a short amount of time, you will be locked from attempting to log in for a short amount of time. Details can be seen here: CheckPasswordStrength()

In local instances of Shuffle, password policies are minimal (8 characters).

MFA and SSO

SAML/SSO and MFA is fully available and can be controlled from an tenant perspective. Users can be a part of multiple tenants, but if even one tenant the user is in enables "MFA Required", it will now be required for the users.

MFA: https://shuffler.io/admin?tab=users

SAML/SSO: https://shuffler.io/admin?admin_tab=sso

Backend API access

There are multiple ways to access the API. The first is through the UI and a logged in user. The second is through the API directly with a Bearer token. The third is from a workflow execution.

• Session Token: Defined in a users' browser as user logs in.
• Bearer Auth: This is a token provided to each user to be used with the API
• Execution token: The execution token is a unique token (UUID) provided to each workflow execution. As soon as an execution is triggered, it gives a temporary token which is valid as long as the workflow is running, but which is no longer valid after the workflow finishes. This has access to certain API's used by Apps themselves (files, cache, setting/changing the execution), with most other API's being off limit.

Session management

Session management in Shuffle is currently quite basic, with the goal to drastically improve it in 2024. The current flowchart includes reusing the same session token across anyone using an account, with the only way to log out everyone else being to log our yourself. The goal is to make this use one session token per device.

App authentication

App authentication is how we authenticate and store an app's configuration. If an app requires authentication, and the user adds the authentication credentials, these credentials will be encrypted and stored in the database, along with being cached in their encrypted form (AES-256). These values can and should NEVER be decrypted to be seen by a process or human other than during a workflow execution.

How are these values being used then? If they're encrypted, how does the app get access to them? Here's how:

1. The workflow starts running. This sets up a lot of different values necessary for Shuffle to find the start node, next nodes, parsing data, fixing conditions etc.
2. The backend looks at ALL nodes in the workflow, checking if any of their parameters have the field "configuration" set to true. This indicates it's a field to be replaced.
3. It finds the appropriate apps' chosen authentication by ID (authentication_id)
4. Shuffle runs decryption on all the appropriate fields, then replaces the value JUST for this workflow execution.
5. The workflow starts with the right values. These values are and should NOT be available to a user reading the workflow at any point.
6. When the app and workflow is finished - all these values are cleaned up and removed from the execution to ensure they're not stored.

Workflow execution model

The execution model of Shuffle can be defined as such:

1. A Workflow is triggered. This can be by a trigger, or a manual execution, making a request towards /api/v1/workflows/<workflow_id>/execute

2. The backend fills in the appropriate gaps (e.g. startnode, source trigger or app authentication), before adding an execution to be retrieved by Orborus at a later stage, with the appropriate priority (0-10, 10 being highest).

3. Orborus runs as an agent, constantly polling for jobs from Shuffle. By default, it retrieves 10 jobs at a time, before checking whether any can be started. This is done based on how many Workers are running in Docker at this time, compared to the environment variable SHUFFLE_ORBORUS_EXECUTION_CONCURRENCY.

   • Orborus also cleans up Docker containers on an interval according to the environment variable SHUFFLE_ORBORUS_EXECUTION_TIMEOUT

4. For each of the new jobs, Orborus creates a container for a Worker with the following information: An execution ID and authorization, and whether to run in an optimized way.

5. A worker is started FOR EACH EXECUTION, which is in control of the entire duration of an execution. It has two modes:

6. Unoptimized: If a workflow is started manually, the Worker will periodically poll the backend for updates, pointing all apps to send direct information to the backend. This makes it possible for the user to see updates in real-time for debug purposes.

7. Optimized: If a Workflow is started from a trigger (not manually), the Worker starts an HTTP server, and acts as a temporary backend for this specific execution. This makes it possible to communicate and deploy Apps faster, without straining the backend. When the workflow is finished, it will send the full execution to the backend. This means the frontend MAY not have the full picture until after the workflow execution finishes.

8. The Worker attempts starting each App's Docker container, starting with the startnode. As it finds that a node has finished, it will check it's status, before starting the following nodes upon success. If the App's Docker image doesn't exist, it will attempt to download in the order of: Backend, Dockerhub. If it doesn't exist, abort the workflow.

9. The apps that were started by a Worker retrieves the full execution in order to be able to identify variables, check conditions, authorization, download files or from the cache etc. Order of operations in an App's Docker container:

   1. Download the workflow execution and current action from the backend.
   2. Check if ALL Conditions are valid or not.
   3. Check each parameter, and whether they contain data that needs replacing ($nodename.json.is.#0.parsed.like_this).
   4. Check Liquid formatting, and replace values with it.
   5. Validate based on the parameters, if it needs to run ONE TIME or MULTIPLE TIMES (for loop).
   6. Run the App! Now that we've parsed all the data, we can finally run the actual app image.
   7. If error from the app, return to the Backend with status "FAILURE" and details on the issue. If success, return status "SUCCESS" and the value of the execution. Logs and timestamps are also added for this.
   8. Exit the Docker container, and self-delete information IF running an optimized Worker.

10. When the Execution in the Worker is in either the FINISHED or ABORTED state, the Worker sends information back to the original Backend about the status of the execution.

To learn about the code behind the execution, check here

Docker container control

As we've hinted at more than a few times, everything in Shuffle is built around Docker containers. This keeps the environment safe even in cases of compromise, and is necessary to allow a user to build an app for pretty much anything. This allows us to sandbox and control how actions are performed, as it's all built on top of our App SDK. On top of all this? It makes sharing and deploying easier between Shuffle users, which is paramount to a scalable, global system.

The most important parts about containerization in Shuffle is this:

• Our App SDK is a Docker container
• If an app is made in Shuffle, either from OpenAPI or from scratch, it's built on top of the App SDK.
• All apps made by and for Shuffle will be released, unless explicitly decided not to.
• All apps are built around the App SDK
• If an app doesn't exist on the Worker's server during an execution, the Worker in question will try to download it from local, then remote registries.

Learn more

Contact us or mail us at help@shuffler.io, and we'll provide you with the necessary information.