Tips and Tricks for data import into SQL Server – Part I
In this post, we will import a .csv into SQL server. While importing csv can be one of the easiest task that can come over to you as a part of your daily duties of being a data engineer, sometimes it can require a bit more thinking and efforts. A similar thing came across, where around 50+ large .csv files in a shared location and had to be imported in SQL server and all of them had varying headers which meant destination tables were supposed to have different schemas. Though, there are numerous methods to achieve this for example you create an SSIS package or may be write a PowerShell script to loop through the files, and import them. However, we looked to use SQL Server’s bulk insert command to complete the task. If you have worked with BULK INSERT, you would have noticed that the destination table must be present beforehand. Therefore, the main challenge is that, destination tables with different schemas have to be created dynamically (on the fly) according to th...