Getting all of users from Firebase Auth with pagination

[evilcore29]

Hi!
I'm trying to getting all of my users from Firebase Auth with pagination. In original case i have more than 1000.

I was trying to use this and this articles.

In first article, provided code triggered infinitely. In the second, i'm getting error like 'Detected infinite loop'. In test config, i'm trying to get users by 1 user per request.

There is no info in the original article about using Firebase auth with pagination. I need just get, store users in query.data and show count of all users from Firebase Auth.

This really simple issue burned more than 6 hours of time. Why is using pagination so confusing? Can anybody tell about how to get more than 1000 users from Firebase Auth with Retool? Or how to use getting users with limit and pagination?

[evilcore29]

With hundreds of attempts, i founded the right way (based in this solution).
I added two params in success trigger to prevent infinite loading users loop.

And just a little modify getAllUsersCode: add if statement to check are there some users in data:

if(data.users.length !== 0) allUsers.setIn([i], data.users)

Full code:

//https://firebase.google.com/docs/auth/admin/manage-users#list_all_users
//firebase auth's list option has a max return of 1000 users. If we want all of them, we'll have to loop through and trigger the query again if we get a pageToken back letting us know we haven't gotten them all yet.

allUsers.setValue([]); //the value of our temp state for storing our baches of users. This must be an array

function next(page, i) {
  getUsers.trigger({
    additionalScope: !!page ? { page: page } : {}, //if we have a token, pass it in the additionalScope
    onSuccess: function (data) {
      if(data.users.length !== 0) allUsers.setIn([i], data.users)
      //onSuccess, save the data into an array in a tempstate at a specific index. 
      if (!!data.pageToken) {
        //if this isn't the final request 
        console.log(data.pageToken);
        next(data.pageToken, i + 1); //re-run getUsers with the pageToken we got, and i+1 to save the next return at in our allUsers temp state
      }
    }
  });
}

next(undefined, 0); //trigger the query the first time, passing in an undefined pageToken

P.S: If this post, see'll someone from Retool Product team, please, improve user experience for standard Firebase modules, such as pagination.

Thank you

[Kabirdas]

Thanks for sharing your solution here @evilcore29! We could definitely make things clearer around pagination for folks. It's on the radar as something to at least have more docs about.

[chawkhsa]

I'm still only able to get 1k rows to return. What I have:
getUsers -- with a success trigger and a next page token value

success handler on getUsers looks like this:

I have a js file that matches your code above named getAllUsers

and I have a variable named allUsers. Is it just supposed to be a null initial value?

Thanks for any help in advance. I still only get 1k users. This is a very frustrating concept in Retool.

I also get this error:

[Kabirdas]

Hey there folks! Want to try jumping in with some alternate ideas about how one might get a full list of users. This query seems to work fairly well:

//this can can change but should be the upper limit for how many pages you expect to retrieve to prevent infinite loops
const pageLimit = 10; 

const allUsers = [];
let currentPageToken = "";

//loop until page limit is reached
for (let i = 0; i < pageLimit; i++) {

  //grab user list an next page token from query
  const { users, pageToken } = await firebaseQuery2.trigger({
    additionalScope: { currentPageToken },
  });

  //aggregate users
  allUsers.push(...users);

  //if there's no new page token the query is done, exit the loop early
  if(!pageToken) break;

  currentPageToken = pageToken;
}

return allUsers;

The corresponding firebase query looks like this:

Note the only configuration is to set currentPageToken that will be passed via additionalScope and to set the number of users to retrieve, no success handlers needed! Also, instead of storing the users in a temp state it should just return the full list as a result of the JS query.

The gif below is of the query running, I set the limit to 2 instead of 1000 because I didn't have many users to paginate through but the same idea should work

Hopefully that helps!