Dynamic queryString in API call

Dynamic URL Construction: Query String Issue with ? being Replaced by %3F

- Goal:
I am trying to dynamically construct a URL by adding query string parameters based on the values filled in certain fields. The expected behavior is to append only the parameters that have a value.

For example:

If item_id = 123 and group_id = 456, the final URL should look like:
https://example.com/api/v1/?page=1&item_id=123&group_id=456.

- Problem:
The issue I'm encountering is that the ? character in the query string is being replaced by %3F when the request is constructed. This results in an incorrect URL like:
https://example.com/api/resource%3Fpage=1&item_id=123&group_id=456.

- Steps Taken So Far:

  1. I created a transformer to dynamically construct the query string based on the fields:
let attributes = [];
let page = 1;

if ({{ item_id.value }}) {
  attributes.push('item_id=' + {{ item_id.value }});
}

if ({{ group_id.value }}) {
  attributes.push('group_id=' + {{ group_id.value }});
}

const final_attributes = attributes.length > 0
  ? '?page=' + page + '&' + attributes.join('&')
  : '?page=' + page;

return final_attributes;

- Details:

  • It seems like Retool is encoding the ? as if it were part of the URL path instead of treating it as the start of a query string.
  • I’ve tried adding the ? directly in the base URL or in the transformer but neither approach has resolved the issue.

- Questions:

  1. How can I ensure that the ? in the query string is not replaced with %3F when Retool constructs the request?
  2. Is there a better way to handle dynamic query string generation in Retool without this issue?

Received API JSON example:

{
  "pagination": {
    "page": 1,
    "limit": 10,
    "totalItems": 10,
    "totalPages": 1,
    "previousPage": 1,
    "nextPage": 2
  },
  "filters": {
    "item": {
      "label": "ITEM",
      "fields": [
        "item_id",
        "item_type"
      ]
    },
    "group": {
      "label": "GRUPO",
      "fields": [
        "group_id",
        "group_type"
      ]
    },
    "order": {
      "label": "ORDER",
      "fields": "order_id"
    },
    "action": {
      "label": "ACTION",
      "fields": "last_action"
    },
    "sender": {
      "label": "SENDER",
      "fields": "sender_id"
    },
    "receiver": {
      "label": "RECEIVER",
      "fields": "recipient_id"
    },
    "price": {
      "label": "PRICE",
      "fields": "price"
    },
    "taxes": {
      "label": "TAXES",
      "fields": [
        "tax_reseller_percentage",
        "fee_producer_percentage",
        "tax_service_percentage",
        "tax_service_fixed"
      ]
    },
    "status": {
      "label": "STATUS",
      "fields": "status"
    },
    "createdAt": {
      "label": "CREATED AT",
      "fields": "created_at"
    },
    "updatedAt": {
      "label": "UPDATED AT",
      "fields": "updated_at"
    },
    "archive": {
      "label": "ARCHIVE",
      "fields": [
        "last_action",
        "reason",
        "archived_at"
      ]
    }
  },
  "items": [
    {
      "resale_id": "7f18f6b1-cbfd-4091-a4c0-297696086219",
      "created_at": "2024-11-19T02:20:24.380Z",
      "updated_at": "2024-11-19T03:14:56.739Z",
      "sender_id": 26203862,
      "recipient_id": 30264748,
      "item_id": "123123123123123",
      "item_type": "ticket",
      "group_id": "93297",
      "group_type": "event",
      "order_id": "123123123123",
      "status": "TRANSFERRING",
      "resale": false,
      "price": null,
      "tax_reseller_percentage": null,
      "fee_producer_percentage": null,
      "tax_service_percentage": null,
      "tax_service_fixed": null,
      "last_action": "TRANSFER_MADE",
      "reason": "transfer completed successfully",
      "archived_at": "2024-11-19T03:15:21.399Z"
    },
    {
      "resale_id": "a186a812-9ea1-442d-96f3-cb31e5c0a249",
      "created_at": "2024-11-18T17:14:38.086Z",
      "updated_at": "2024-11-18T18:19:27.417Z",
      "sender_id": 1851170,
      "recipient_id": 19772886,
      "item_id": "123123123123123",
      "item_type": "ticket",
      "group_id": "93338",
      "group_type": "event",
      "order_id": "123123123123",
      "status": "TRANSFERRING",
      "resale": false,
      "price": null,
      "tax_reseller_percentage": null,
      "fee_producer_percentage": null,
      "tax_service_percentage": null,
      "tax_service_fixed": null,
      "last_action": "TRANSFER_MADE",
      "reason": "transfer completed successfully",
      "archived_at": "2024-11-18T18:19:51.650Z"
    }
  ]
}

Hi Maicol,

This is just a guess, but you might be able to avoid the character encoding issue by specifying the parameters explicitly as URL parameters individually.

You may need to adjust the structure of the final_attributes that your transformer returns in order to individually reference them as URL parameters. (One approach would be to return an object with param names as keys which you reference in the query.)

Screenshot 2024-11-19 at 9.52.27 AM

Thank you for your response!

I understand that explicitly referencing the parameters could solve the encoding issue, but I was wondering if there’s a way to dynamically add the fields to the query string based on whether they have a value or not.

For example, in my use case:

  • If item_id has a value, it should be added as item_id=value.
  • If group_id also has a value, it should be appended as group_id=value.
  • If neither has a value, no query string should be generated except for the default page=1.

Additionally, I would like to ask:

  • Is it possible to create a transformer that dynamically constructs the query string, considering that pagination (page) and ordering are fully controlled by the API?
  • Specifically, whenever a filter is applied, the page attribute needs to reset to 1 or remain as the current page but with the filter applied. Is this achievable through a Retool transformer?

I’m trying to ensure the query string dynamically reflects the current state of the filters, pagination, and ordering without breaking the API’s expected behavior. Any guidance or examples would be greatly appreciated!

ahh, thanks for the clarification!

So this is not pretty, but I think this could yield what you're looking for: including or omitting the param based on the presence (or type-correctness) of a value:

(Excuse the red; I'm looking at an example from our Query Library so there are bad references.)

This accomplishes what I need in certain contexts, such as when an endpoint doesn't support having a param name with a null value, i.e. it omits the entire key/value.

Not sure if it helps with your additional asks but it might get you one step closer :slight_smile:

2 Likes

Agreed with that suggestion! Otherwise, worst case, I'd try removing the base url from the resource page & constructing the whole url dynamically

Your approach worked, but I was wondering if there is another way to filter, paginate, and sort the API response without mapping input by input. Another thing is that when I populate a field for filtering, the page variable needs to be reset to 1. I need to automate this because I have many fields to map. When I find a way to do this, I’ll post it here.

1 Like

When I try to build a URL with a query parameter, the "?" character gets encoded as "%3F." I'm trying to create a system that doesn't manage the data directly; all filtering, pagination, and ordering should be handled by the API. Did you have a chance to look at the response body I sent? Do you think there's any tool that could help me with this? I feel like I'm trying to solve something that should be straightforward.

Hi @Maicol_Oliveira Do you mean this is happening when you try to clear the baseurl and create the entire url in the app? Could you share a screenshot?

Hi Tess, as requested, please find below the screenshots of the test showing the character conversion issue.

Got it! Thanks

If you keep the ? separate from the transformer value, it won't get encoded as %3F. Unfortunately, we have a bug where this approach automatically appends a trailing = which has caused issues for some apis

You are correct. When I tested leaving the "?" out of the transformer, it remained intact. However, in the end, the "=" sign was included. When this symbol reaches my API, I encounter issues as the framework does not accept it.

The company has requested to hold off on the implementation for now until we can optimize its usage. I will continue working on the implementation next year. If I keep facing issues, I'll open another thread here.

Thank you very much for your support!