Today

S3


S3

S3

Beta version

The configuration and connection might need a bit of fine-tuning to get things running smoothly. Let us know if you hit any snags at team@tably.com.

Once created, a connection can be reused or copy-pasted in all other pages within same workspace without additional setup. The setup can be complex and, if in doubt, ask your developer for help. They will have all needed details and will be able to assist with possible errors.

The name of the stream you would like this source to output. Can contain letters, numbers, or underscores.
A regular expression which tells the connector which files to import. All files which match this pattern will be imported. Use | to separate multiple patterns. See this page to understand pattern syntax (GLOBSTAR and SPLIT flags are enabled). Use pattern ** to pick up all files.

Use this to load files from S3 or S3-compatible services

Name of the S3 bucket where the file(s) exist.
In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.
In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.
By providing a path-like prefix (e.g. myFolder/thisTable/) under which all the relevant files sit, we can optimize finding these in S3. This is optional but recommended if your bucket contains many folders/files which you don't need to import.
Endpoint to an S3 compatible service. Leave empty to use AWS.
The format of the files you'd like to import

This connector utilises [PyArrow (Apache Arrow)](https: // arrow.apache.org/docs/python/generated/pyarrow.csv.open_csv.html) for CSV parsing.

The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
Configures whether a schema for the source should be inferred from the current data or not. If set to false and a custom schema is set, then the manually enforced schema is used. If a schema is not manually set, and this is set to false, then all fields will be read as strings
The character used for quoting CSV values. To disallow quoting, make this field blank.
The character used for escaping special characters. To disallow escaping, leave this field blank.
The character encoding of the CSV data. Leave blank to default to UTF8. See list of python encodings for allowable options.
Whether two quotes in a quoted CSV value denote a single quote in the data.
Whether newline characters are allowed in CSV values. Turning this on may affect performance. Leave blank to default to False.
Optionally add a valid JSON string here to provide additional options to the csv reader. Mappings must correspond to options detailed here. 'column_types' is used internally to handle schema so overriding that would likely cause problems.
Optionally add a valid JSON string here to provide additional Pyarrow ReadOptions. Specify 'column_names' here if your CSV doesn't have header, or if you want to use custom column names. 'block_size' and 'encoding' are already used above, specify them again here will override the values above.
The chunk size in bytes to process at a time in memory from each file. If your data is particularly wide and failing during schema detection, increasing this should solve it. Beware of raising this too high as you could hit OOM errors.

This connector utilises PyArrow (Apache Arrow) for Parquet parsing.

If you only want to sync a subset of the columns from the file(s), add the columns you want here as a comma-delimited list. Leave it empty to sync all columns.
Maximum number of records per batch read from the input files. Batches may be smaller if there aren’t enough rows in the file. This option can help avoid out-of-memory errors if your data is particularly wide.
Perform read buffering when deserializing individual column chunks. By default every group column will be loaded fully to memory. This option can help avoid out-of-memory errors if your data is particularly wide.

This connector uses PyArrow for JSON Lines (jsonl) file parsing.

Whether newline characters are allowed in JSON values. Turning this on may affect performance. Leave blank to default to False.
How JSON fields outside of explicit_schema (if given) are treated. Check PyArrow documentation for details
The chunk size in bytes to process at a time in memory from each file. If your data is particularly wide and failing during schema detection, increasing this should solve it. Beware of raising this too high as you could hit OOM errors.
Optionally provide a schema to enforce, as a valid JSON string. Ensure this is a mapping of { "column" : "type" }, where types are valid JSON Schema datatypes. Leave as {} to auto-infer the schema.
Finish step configuration to see output