RemoteStorageApi¶

class RemoteStorageApi(api)[source]¶

Bases: ModuleApiBase

API for listing and downloading files from remote/cloud storages connected to a team.

Parameters:

api¶: Api object to use for API connection.

Methods

`convert_info_to_json`	Convert information about an entity to a dictionary.
`download_path`	Downloads item from given remote path to given local path.
`get_file_info_by_path`	Get info about file for given remote path.
`get_info_by_id`	Get information about an entity by its ID from the Supervisely server.
`get_list_all_pages`	Get list of all or limited quantity entities from the Supervisely server.
`get_list_all_pages_generator`	This generator function retrieves a list of all or a limited quantity of entities from the Supervisely server, yielding batches of entities as they are retrieved
`get_list_available_providers`	Get the list of available providers for the instance.
`get_list_idx_page_async`	Get the list of items for a given page number.
`get_list_page_generator_async`	Yields list of images in dataset asynchronously page by page.
`get_list_supported_providers`	Get the list of supported providers for the instance.
`get_path_stats`	Get information about file size and the date of its last modification in bucket storage.
`get_remote_path`	Returns remote path.
`info_sequence`	Get list of all class field names.
`info_tuple_name`	Get string name of NamedTuple.
`is_bucket_url`	Check if the URL is a bucket URL.
`is_path_exist`	Check if the file path exists.
`list`	List files and directories for given remote path.
`upload_path`	Uploads item from given local path to given remote path.

Attributes

`MAX_WAIT_ATTEMPTS`	Maximum number of attempts that will be made to wait for a certain condition to be met.
`WAIT_ATTEMPT_TIMEOUT_SEC`	Number of seconds for intervals between attempts.

classmethod convert_info_to_json(info)¶

Convert information about an entity to a dictionary.

Parameters:

info : NamedTuple¶: Information about the entity.

Returns:

Dictionary with information about the entity.

Return type:

dict

static info_sequence()¶: Get list of all class field names.

static info_tuple_name()¶: Get string name of NamedTuple.

static is_bucket_url(url)[source]¶

Check if the URL is a bucket URL.

Parameters:

url : str¶: URL

Returns:

True if URL is a bucket URL, False otherwise

Return type:

bool

Usage Example:

from supervisely.api.remote_storage_api import RemoteStorageApi

url = "s3://bucket/lemons/ds1/img/IMG_444.jpeg"
RemoteStorageApi.is_bucket_url(url)

download_path(remote_path, save_path, progress_cb=None, team_id=None)[source]¶

Downloads item from given remote path to given local path.

Parameters:

remote_path : str¶: Remote path to item that you want to download.
save_path : str¶: Local save path.
progress_cb : tqdm or callable, optional¶: Progress function to download.
team_id : int¶: Team ID (to get cloud storages connected to the team)

provider = "s3" # can be one of ["s3", "google", "azure"]
bucket = "bucket-test-export"
path_in_bucket = "/demo/image.jpg"
remote_path = api.remote_storage.get_remote_path(provider, bucket, path_in_bucket)
# or alternatively use this:
# remote_path = f"{provider}://{bucket}{path_in_bucket}"
api.remote_storage.download_path(local_path="images/my-cats.jpg", remote_path=remote_path)

get_file_info_by_path(path, team_id=None)[source]¶

Get info about file for given remote path.

Parameters:

path : str¶: Remote path to file.
team_id : int¶: Team ID (to get cloud storages connected to the team)

Returns:

file info in the given remote path

Return type:

Optional[dict]

get_info_by_id(id)¶

Get information about an entity by its ID from the Supervisely server.

Parameters:

id : int¶: ID of the entity.

get_list_all_pages(method, data, progress_cb=None, convert_json_info_cb=None, limit=None, return_first_response=False)¶

Get list of all or limited quantity entities from the Supervisely server.

Parameters:

method : str¶: Request method name
data : dict¶: Dictionary with request body info
progress_cb : Progress, optional¶: Function for tracking download progress.
convert_json_info_cb : Callable, optional¶: Function for convert json info
limit : int, optional¶: Number of entity to retrieve
return_first_response : bool, optional¶: Specify if return first response

Returns:

List of entities.

Return type:

List[dict]

get_list_all_pages_generator(method, data, progress_cb=None, convert_json_info_cb=None, limit=None, return_first_response=False)¶

This generator function retrieves a list of all or a limited quantity of entities from the Supervisely server, yielding batches of entities as they are retrieved

Parameters:

method : str¶: Request method name
data : dict¶: Dictionary with request body info
progress_cb : Progress, optional¶: Function for tracking download progress.
convert_json_info_cb : Callable, optional¶: Function for convert json info
limit : int, optional¶: Number of entity to retrieve
return_first_response : bool, optional¶: Specify if return first response

get_list_available_providers(team_id=None)[source]¶

Get the list of available providers for the instance.

Parameters:

team_id : int¶: Team ID (to get cloud storages connected to the team)

Returns:

List of available providers

Return type:

List[dict]

Usage Example:

import os
from dotenv import load_dotenv
import supervisely as sly

# Load secrets and create API object from .env file (recommended)
# Learn more here: https://developer.supervisely.com/getting-started/basics-of-authentication
if sly.is_development():
    load_dotenv(os.path.expanduser("~/supervisely.env"))

api = sly.Api.from_env()

team_id = 123
available_providers = api.remote_storage.get_list_available_providers(team_id)

# Output example
#    [
#        {
#            "id": "minio",
#            "name": "Amazon S3",
#            "defaultProtocol": "s3:",
#            "protocols": [
#                "s3:",
#                "minio:"
#            ],
#            "buckets": [
#                "bucket-test",
#                "remote-img"
#            ]
#        }
#    ]

async get_list_idx_page_async(method, data)¶

Get the list of items for a given page number. Page number is specified in the data dictionary.

Parameters:

method : str¶: Method to call for listing items.
data : dict¶: Data to pass to the API method.

Returns:

List of items.

Return type:

Tuple[int, List[NamedTuple]]

async get_list_page_generator_async(method, data, pages_count=None, semaphore=None)¶

Yields list of images in dataset asynchronously page by page.

Parameters:

method : str¶: Method to call for listing items.
data : dict¶: Data to pass to the API method.
pages_count : int, optional¶: Preferred number of pages to retrieve if used with a per_page limit. Will be automatically adjusted if the pagesCount differs from the requested number.
semaphore=None¶: Semaphore for limiting the number of simultaneous requests.

Returns:

List of images in dataset.

Return type:

AsyncGenerator[List[ImageInfo]]

Usage Example:

import os
from dotenv import load_dotenv

import supervisely as sly

# Load secrets and create API object from .env file (recommended)
# Learn more here: https://developer.supervisely.com/getting-started/basics-of-authentication
if sly.is_development():
    load_dotenv(os.path.expanduser("~/supervisely.env"))

api = sly.Api.from_env()

method = 'images.list'
data = {'datasetId': 123456}

loop = sly.utils.get_or_create_event_loop()
images = loop.run_until_complete(api.image.get_list_generator_async(method, data))

get_list_supported_providers(team_id=None)[source]¶

Get the list of supported providers for the instance.

Parameters:

team_id : int¶: Team ID (to get cloud storages connected to the team)

Returns:

List of supported providers

Return type:

List[dict]

Usage Example:

import os
from dotenv import load_dotenv

import supervisely as sly

# Load secrets and create API object from .env file (recommended)
# Learn more here: https://developer.supervisely.com/getting-started/basics-of-authentication
if sly.is_development():
    load_dotenv(os.path.expanduser("~/supervisely.env"))

api = sly.Api.from_env()

team_id = 123
supported_providers = api.remote_storage.get_list_supported_providers(team_id)

# Output example
#    [
#        {
#            "id": "google",
#            "name": "Google Cloud Storage",
#            "defaultProtocol": "google:",
#            "protocols": [
#                "google:",
#                "gcs:"
#            ]
#        }
#    ]

get_path_stats(path, team_id=None)[source]¶

Get information about file size and the date of its last modification in bucket storage.

Parameters:

path : str¶: URL of the file in the bucket storage
team_id : int¶: Team ID (to get cloud storages connected to the team)

Returns:

File ‘size’ in bytes and ‘lastModified’ date if file exists, otherwise None

Return type:

Optional[dict]

Usage Example:

import os
from dotenv import load_dotenv

import supervisely as sly

# Load secrets and create API object from .env file (recommended)
# Learn more here: https://developer.supervisely.com/getting-started/basics-of-authentication
if sly.is_development():
    load_dotenv(os.path.expanduser("~/supervisely.env"))

api = sly.Api.from_env()

path = "s3://bucket/lemons/ds1/img/IMG_444.jpeg"
team_id = 123
stats = api.remote_storage.get_path_stats(path, team_id)

# Output example

#   {
#       "size": 155790,
#       "lastModified": "2023-01-26T09:20:27.000Z"
#   }

get_remote_path(provider, bucket, path_in_bucket)[source]¶

Returns remote path.

Parameters:

provider : str¶: Can be one of “s3”, “google”, “azure”.
bucket : str¶: Name of the bucket container.
path_in_bucket : str¶: Path to item in bucket.

Usage Example:

import os
from dotenv import load_dotenv

import supervisely as sly

# Load secrets and create API object from .env file (recommended)
# Learn more here: https://developer.supervisely.com/getting-started/basics-of-authentication
if sly.is_development():
    load_dotenv(os.path.expanduser("~/supervisely.env"))

api = sly.Api.from_env()

provider = "s3"
bucket = "bucket-test-export"
path_in_bucket = "/demo/image.jpg"
remote_path = api.remote_storage.get_remote_path(provider, bucket, path_in_bucket)
# Output: s3://bucket-test-export/demo/image.jpg

is_path_exist(path, team_id=None)[source]¶

Check if the file path exists.

Parameters:

path : str¶: URL of the file in the bucket storage
team_id : int¶: Team ID (to get cloud storages connected to the team)

Returns:

True if the file exists, False otherwise

Return type:

bool

Usage Example:

import os
from dotenv import load_dotenv

import supervisely as sly

# Load secrets and create API object from .env file (recommended)
# Learn more here: https://developer.supervisely.com/getting-started/basics-of-authentication
if sly.is_development():
    load_dotenv(os.path.expanduser("~/supervisely.env"))

api = sly.Api.from_env()

path = "s3://bucket/lemons/ds1/img/IMG_444.jpeg"
is_exist = api.remote_storage.is_path_exist(path)

list(path, recursive=True, files=True, folders=True, limit=10000, start_after='', team_id=None)[source]¶

List files and directories for given remote path.

Parameters:

path : str¶: Remote path with items that you want to list.
recursive : bool¶: List remote path revursively.
files : bool¶: List files in the given path.
folders : bool¶: List folders in the given path.
limit : int¶: Limit of files to list. 10000 is the maximum limit.
start_after : str¶: Start listing path after given file name.
team_id : int¶: Team ID (to get cloud storages connected to the team)

Returns:

List of files in the given remote path

Return type:

list

upload_path(local_path, remote_path, team_id=None)[source]¶

Uploads item from given local path to given remote path.

Parameters:

local_path : str¶: Local path to item that you want to upload.
remote_path : str¶: Remote destination path.
team_id : int¶: Team ID (to get cloud storages connected to the team)

Usage Example:

import os
from dotenv import load_dotenv

import supervisely as sly

# Load secrets and create API object from .env file (recommended)
# Learn more here: https://developer.supervisely.com/getting-started/basics-of-authentication
if sly.is_development():
    load_dotenv(os.path.expanduser("~/supervisely.env"))

api = sly.Api.from_env()

provider = "s3" # can be one of ["s3", "google", "azure"]
bucket = "bucket-test-export"
path_in_bucket = "/demo/image.jpg"
remote_path = api.remote_storage.get_remote_path(provider, bucket, path_in_bucket)
team_id = 123
# or alternatively use this:
# remote_path = f"{provider}://{bucket}{path_in_bucket}"
api.remote_storage.upload_path("images/my-cats.jpg", remote_path, team_id)