[how-to] Upload large files to S3

Retool has about a 50MB upload limit using its various File components. You can use fetch() to send larger files, but you have to convert them to blobs first and there are practical limits on file sizes there as well.

The solution is to use the browser's File Input type. But for that you need a Custom component and they can be difficult to integrate into Retool; getting it to look like the same UI and doing the proper communications between Custom components and Retool.

This tutorial shows you how to accomplish this.

Here is the finished app. You can't even tell that there is a Custom component in there.

First we need to add the custom component, called ccUploadtoS3, and throw in this code. it is heavily commented so you can follow what it is doing:

<style>
  * {
    font-family: 'Inter', sans-serif;
    font-weight: 500;
    font-size: 12px;
    line-height: 16px;
  }

  body {
    margin: 0;
  }

  .button-container {
    display: flex;
    align-items: center;
    gap: 8px;
    margin-top: 12px;
  }

  .button-disabled {
    background-color: #bbbbbb;
    color: #ffffff;
    cursor: not-allowed !important;
  }
  
  .button {
    border: none;
    border-radius: 4px;
    padding: 8px 16px;
    font-weight: 500;
    font-size: 12px;
    cursor: pointer;
    outline: none;
    appearance: none;
    user-select: auto;
  }

  .button-main {
    background-color: #3C92DC;
    color: #ffffff;
  }

  .button-main:hover {
    background-color: #4DA3ED;
  }

</style>

<!-- This is the HTML or the component -->
<div class="button-container">
  <button class="button button-main"onclick="document.getElementById('file-input').click()" id="select-button">Step 1. Select a File</button>
  <input  class="button button-main" value="Step 1." type="file" id="file-input"  style="display:none"/>
  <button class="button button-disabled" id="prep-button">Step 2. Prepare File</button>
  <button class="button button-disabled" id="upload-button">Step 3. Upload</button>
  <button class="button button-disabled" id="reset-button">Reset</button>
</div>

<script type="text/babel">
// Throw all of our DOM elements into variables to make them a tad easier to use later on.
const fileInput = document.getElementById("file-input");
const prepButton = document.getElementById("prep-button");
const uploadButton = document.getElementById("upload-button");
const resetButton = document.getElementById("reset-button");
const selectButton = document.getElementById("select-button");
const log = document.getElementById("log");
const reader = new FileReader();
// Get a hook into the Model and inject model.signedUrl into the uploader
let uploader;
window.Retool.subscribe((model) => uploader = uploadData.bind(null, model.signedUrl));

// Change Handler for the File input
fileInput.addEventListener("change", () => {
  // Load the file into memory
  reader.readAsArrayBuffer(fileInput.files[0])
  // Send the file name back to Retool
  updateFileName(fileInput.files[0].name)
  // Update the Status so the user is informed
  updateStatus("Ready to Prep File")
  // Manage Button state
  disableButton(selectButton)
  enableButton(resetButton)
  enableButton(prepButton)  
}); 

prepButton.addEventListener("click", () => {
  // Run query that gets S3's signed URL
  window.Retool.triggerQuery('getSignedURL')
  // Update the Status so the user is informed
  updateStatus("Ready to Upload")
// Manage Button state  
  disableButton(prepButton)
  enableButton(uploadButton)
});

// Click handler for the Upload button
uploadButton.addEventListener("click", () => {
  // Double check we have a file to upload (optional)
  if (fileInput.files.length > 0) {
    // Update the Status do the user is informed
    updateStatus("Uploading...")
    // Upload the file to S3
    uploader(reader.result, fileInput.files[0].type);
    // Manage Button state
    disableButton(uploadButton)
  }
});

// Click handler for the reset button
resetButton.addEventListener("click", () => {
  // Clear File input
  fileInput.value = null
  // Tell Retool there is no longer a file in play
  updateFileName(null)
  // Update the Status do teh user is informed
  updateStatus("")
  // Manage Button state
  disableButton(resetButton)
  enableButton(selectButton)
  disableButton(prepButton)
  disableButton(uploadButton)
});

async function uploadData (url, data, type) {
  // url was injected earlier and comes from the getSignedUrl query
  const request = new Request(url, {
    method: "PUT",
    headers: {
      "Content-Type": type,
    },
    body: data,
  });
  // This does the uploading
  const response = await fetch(request);
  // Update the Status do the user is informed
  updateStatus("Uploading is Complete...")
}

function updateStatus(status) {
  // Updates the model so that Retool can see it.
  window.Retool.modelUpdate({ status: status})
  // Fires a query over in Retool that updates the status
  window.Retool.triggerQuery('onStatusChange')
}

function updateFileName(fileName) {
  // Same as above, we cannot send additionalScope to the query from a Custom component
  // so this technique is a workaround. See onFileNameChange query for the rest of the story
  window.Retool.modelUpdate({ filename: fileName })
  window.Retool.triggerQuery('onFileNameChange')
}

function enableButton(btn) {
  //Just some DOM stuff to change the button style
  btn.disabled=false
  btn.classList.remove("button-disabled");
  btn.classList.add("button-main");
}

function disableButton(btn) {
  btn.disabled=true
  btn.classList.add("button-disabled");
  btn.classList.remove("button-main");  
}

</script>

This is the Modal to use:

{
  "signedUrl": {{getSignedURL?.data === null ? "" : getSignedURL.data.signedUrl}},
  "filename": "",
  "status": ""
}

Now add two temp state vars, currentFile and uploadStatus.

We need two communication handlers. These are triggered within ccUploadtoS3 to send data back to Retool.

Make one called onFileNameChange with is one line of code

currentFile.setValue(ccUploadtoS3.model.filename)

And make one called onStatusChange.

uploadStatus.setValue(ccUploadtoS3.model.status)

Drop in Text component that will show you the file name. Set its value to ##### {{currentFile.value}}.

And we need some way to get status back from ccUploadtoS3, so add an Alert component and set its title to Upload Status and its Description to {{uploadStatus.value}}

We need a query that will get the Signed URL from S3 which is needed to send a file there (don't ask me why, just following directions!) You will need to properly set up the S3 Resource, but that is beyond the scope of this tutorial - there are other resources for that. But the query looks like this:

Finally, assuming you will be doing something after the file is uploaded add a Button and set its Disabled property to {{!uploadStatus.value.includes('Complete')}}. This will enable the button only if ccUploadtoS3 has uploaded the file. Add whatever handler to the button you need for your application.

You now should have a fully functional S3 uploader with some UI tricks.

  1. Click the Select a File button
  • A file picker will pop up for you to select a file.
  • The file name is sent back to Retool.
  • The file name is displayed.
  • Status will update
  • The Select button will disable
  • The Prepare button will enable.
  • Reset button will enable
  1. Click the Prepare button.
  • S3 is prepped for the file.
  • Status will update
  • The Prepare button will disable
  • The Upload button will enable.
  1. Now you click the Upload button.
  • Status will update.
  • File will upload.
  • Status will update again.
  • Upload Button will disable.
  • Process File button will enable.
  1. You can click the Reset button any time after the file selection and all will reset and roll back to the beginning of the cycle.

Possible Improvements

I would really like to be able to run getSignedURL when the file is selected like so:

fileInput.addEventListener("change", () => {
  reader.readAsArrayBuffer(fileInput.files[0])
  updateFileName(fileInput.files[0].name)
  window.Retool.triggerQuery('getSignedURL')
}); 

The problem is the file name is not updated in Retool fast enough and getSignedURL will be run with no, or the previously used, file name. I tried a sprinkling of Async/Await but I can't seem to get things reliably working.

2 Likes

Hey @bradlymathews Awesome walkthrough! I'm trying to get something similar working for myself. I'm a little stuck at this bit:

// onFileNameChange
currentFile.setValue(ccUploadtoS3.model.filename)

// onStatusChange
uploadStatus.setValue(ccUploadtoS3.model.status)

Where are those supposed to go?

Also, what should the custom component model area look like?

Oh, I forgot to put the model in there! Thanks for catching it, I will edit to add that.

Those are the names of the names of the two js queries you need to make. I will clarify that as well.

@bradlymathews Awesome, that did the trick. Thanks for saving me hours of tinkering!