Data Streaming

Users can now stream data from their cloud buckets directly into the app without having to upload it. Whatever state of files users have in the bucket is automatically synced with the corresponding files in users datasets. With this now, users can also add or delete files from datasets by simply adding/deleting them in your bucket.

Users can create a Data stream by clicking on create Data stream button on the data streams page.

Images and Point clouds(JSON Format) are supported as of now, and we are working on adding more file type supports.

User needs to choose which cloud provider they need and then provide few details which are bucket name(bucket where your data is stored), folder name(location of the folder that will be synced to create a dataset present within the bucket), credential file(a json file with the private key of one user that can enable us to authenticate to your bucket location) and optionally user can provide a dataset configuration profile which will be applied to all the datasets that are created using this data stream configuration.

Once a data stream configuration is created, you can click on the "Sync bucket" option to sync your datasets with your bucket instantly. Post that you can use this dataset like normal datasets and it has all the native functionalities that are already supported by Datasets.