Retool Database (PostgreSQL) - Data upload for large CSV file

Discussion
1

Hello all,

we are working currently on the dataset to upload to the Retool Postgres Database. Or background and SQL skills are rather junior.
However, our dataset is a CSV >600 MB and I cannot imagine that creating & uploading 200 * 3 MB CSV files is intended.

How are you handling this? We followed this approach but then need to append 199 tables to the initial table.

Can you point me to any other (simpler) solution.

Thank you very much!

2

From Discord:

  1. Ses — Today at 2:19 PM

well funnily enough, 600MB of data is fairly trivial. This amount of data is not at all uncommon, and can be handled in a few different ways. When you say "then need to append 199 tables to the initial table" -- can you explain what you mean?

1

  1. @Ses

well funnily enough, 600MB of data is fairly trivial. This amount of data is not at all uncommon, and can be handled in a few different ways. When you say "then need to append 199 tables to the initial table" -- can you explain what you mean?

rubicscube11 — Today at 2:22 PM

I figured it is our inability to grasp the problem
Following this approach and uploading 4 files resulted in 4 new tables, which needed to be appended. Insert CSV data into a database (edited)

Retool Docs

Insert CSV data into a database

Learn how to build an app that writes a CSV file to a database.

Insert CSV data into a database

  1. @rubicscube11

I figured it is our inability to grasp the problem Following this approach and uploading 4 files resulted in 4 new tables, which needed to be appended. Insert CSV data into a database (edited)

Ses — Today at 2:26 PM

this kind of tutorial is more for if you need an app you're creating to support uploading CSVs If you're just trying to load data into the retool DB, you can find the db's connection string here:

Image

  1. [2:29 PM]

and then you can use the myriad of ways to load csv data into a table Postgres has a built in COPY way to import CSVs into a table if you can connect with psql: How to import CSV file data into a PostgreSQL table - Stack Overflow And if that doesn't work, there are a few sql clients that can do it too: Import/Export options from CSV to database - Features | DataGrip (Jetbrains Datagrip for example)