Learn how to automate a daily data transfer process on Windows, from a PostgreSQL database to a remote server
The process of transferring files from one location to another is obviously a perfect candidate for automation. It can be overwhelming to do it repeatedly, especially when you have to perform the entire ETL (Extract, Transform, Load) process for multiple sets of data.
Imagine your company has its data in its data warehouse and then decides to outsource some of its analytics to an external data analytics provider. This provider offers bespoke analytics software that will display dashboards and reports for your company’s central production team.
What this means is that you, as a data engineer, will have to transfer data to this provider daily, hourly, every 30 minutes or any other frequency decided by the outsourcing contract.
This article explains in detail this ETL process that includes an SFTP upload. We will incorporate the Secure File Transfer Protocol (SFTP), which is a secure means of transferring files between two remote servers, encrypting the files using what is known as the Secure Shell (SSH) protocol.
