I have an requirement to load hadoop snappy compressed avro file to Big query.
Saw from Google docs that big query detects snappy compression. But when I tried
bq load --source-format=AVRO project:dataset.table gs://mybucket/inputsnappy.snappy
I got "Apache Avro library failed to parse the header with the following error: Invalid data file. Magic does not match" error.
Any input on this will really help.
Also Google doc says only compression on data blocks can be detected by bigquery. Can some one help me understand that point on data blocks.
I also tried converting snappy to avro using python snappy. But iam getting error when doing "decompressed_data= snappy.decompress(input_data)"
Error :Uncompress:invalid input file.
Not sure how to proceed now.