How do i set different datasets for the different environments and making sure that when published the production doesn't get overwritten by staging dataset. I am using Big Query.
2 Likes
Hey @software_cc, and welcome to the forum!
What have you tried so far?
I find this guide a very good first step, in case you hadn't seen it.
1 Like
I'll echo @MiguelOrtiz's welcome and recommendation, @software_cc! The documentation is definitely a good place to start. 
Don't hesitate to follow up if you have any additional questions, though.
To manage different datasets in BigQuery across environments (dev, staging, production), create separate datasets like project.dev_dataset, project.staging_dataset, and project.prod_dataset. Use environment-specific variables or configurations in your CI/CD pipeline to control which dataset is used. When deploying, ensure only non-production environments are allowed to write to dev or staging datasets. Implement access controls to restrict staging from writing to production. Use aliases or views in development for testing, but point them to actual production tables during release. Always double-check deployment scripts to prevent overwriting production data. Backup production regularly for added safety.
Have you had a chance to review this, @KamalDeepPareek? Let us know if you have any additional questions!