Share Data Without Sharing Credentials: Introducing Pipe-level Permissions

Throughout the Flex.io journey, we’ve often used the phrase: “Data is a Team Sport.”

More often than not data processing requires some type of hand off between an ‘owner’ and a ‘user’. For instance, a business user might need an extract from an IT-supported system. Or a consultant may need a file that is owned by her customer.

One party has the permissions. The other party uses the information.

Take My Data… Please.

To help bring owners and users together to share data, we’ve just launched a streamlined way to manage permissions at the pipe-level.

These pipe-level permissions enable a data owner to set up a flow with their credentials. Then they can share that flow with an end user, without giving access to those credentials.

For example, let’s say a business user wants to get a regular extract from your MySQL database. You could give them permissions, but maybe they don’t know how to handle a SQL statement. Alternatively you could go the more common route of sending a CSV file by email, but plain text over the interwebs is not secure.

Or, you could could set up a flow in a couple minutes from your MySQL database to their private Dropbox folder. Your credentials allow access to the data. Their credentials allow access to Dropbox. The pipe acts as the broker between the two systems. And credentials are never exposed to the other party.

Getting Optimized Website Images into an AWS S3 Bucket

One way we use this functionality in-house is for handling website images. In this case, the business user is passing files to the developer’s system.

Specifically, our web documentation has a bunch of screesnhots that we host on S3. I, personally, wrote a bunch of this documentation, but, as a non-dev, I have no S3 rights. So how do I get the images to S3?

Well, there are workarounds — for instance, I could be given rights and then authenticate with a desktop tool that allows me transfer files. Or I could just put them in a folder and have a developer do the transfer (they have nothing else to do, right?).

Or, I could just share data with a simple pipe, on-demand. In this case, the developer set up a pipe that transferred images into the specific bucket we’re using for web assets. He shared the pipe with me. Then I used our Command Line Interface to simply push images as I created them.

This has worked great as a transfer mechanism to share data. Our next step is to add a little Python code to optimize the image file size (site load speed!) and we’ll be in business. I’ll add a how-to post for this pipe in the near future.

What Say Ye, Projects?

Up until recently, our notion of sharing was at the “project” level, meaning you had to set up a project and invite people into it. Every invitee would have broad permissions over everything in your project. In addition to a heavy-handed administrative layer, project-centric sharing made it really difficult to share, copy or organize individual flows.

So, as part of our move to pipe-level sharing, we’ve eliminated the concept of projects for now. That said, organizing flows is still quite important, so expect upgrades in this area coming soon to theater near you. It’ll just be in the context of a simpler, flatter pipe list.

If you have any questions, shoot us a note or contact us via the intercom chat in the app.

Until then, happy pipe sharing!