31 - Using Meltano to load data into DuckDB

Last updated Nov 1, 2022 Edit Source

Daily log

I used Meltano, specifically the tap-spreadsheets-anywhere extractor to load data into DuckDB using the target-duckdb loader. It worked great for my Opportunity Atlas data, but unfortunately was unable to find the Zillow data or the Census-tract-to-zipcode data that I was also hoping to load.

I was also hoping to get to the dbt work, but that will have to wait for another time. I stopped early because I had a dentist’s appointment.

Here is the error I saw:

1
2
3
4
5


2022-11-01T13:23:49.956476Z [info     ] INFO Walking data.             cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2022-11-01T13:23:49.956642Z [info     ] INFO Found 7 files.            cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2022-11-01T13:23:49.956867Z [info     ] INFO Checking 7 resolved objects for any that match regular expression "ZIP_TRACT_*.xlsx" and were modified since 2017-05-01 00:00:00+00:00 cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2022-11-01T13:23:49.957088Z [info     ] INFO Processing 0 resolved objects that met our criteria. Enable debug verbosity logging for more details. cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere
2022-11-01T13:23:49.957331Z [info     ] INFO Wrote 0 records for stream "zip_tract_mapping". cmd_type=elb consumer=False name=tap-spreadsheets-anywhere producer=True stdio=stderr string_id=tap-spreadsheets-anywhere

The filename is data/ZIP_TRACT_122021.xlsx, and I am using the fuzzy match pattern ZIP_TRACT_*.xlsx. Fortunately, I found that I could just specify the filename exactly and I would have an issue. If I continue with this project and need to use it for constantly refreshing data then this might turn into a problem.

I'm a freelance software developer located in Denver, Colorado. If you're interested in working together or would just like to say hi you can reach me at me@ this domain.

🐙 Evan Azevedo

31 - Using Meltano to load data into DuckDB

Backlinks

Interactive Graph