Shell script to concatenate multiple csv files

I had a unix directory with multiple CSV files, all with the same prefix.  I needed them to be concatenated together to make one large file using a shell script.

I used the below code, copied multiple times (one for each file prefix)

Sample files:
file20170804.csv
file20170803.csv
file20170802.csv


new_file=”/dir/file.csv”
if [ -f $new_file ] ; then
rm $new_file
fi

for file in /dir/file?*.csv
do
tail -n +2 “$file” >> $new_file
done


Here is the code with comments:

1 new_file=”/dir/file.csv”
2 if [ -f $new_file ] ; then
3 rm $new_file
4 fi

5 for file in /dir/file?*.csv
6 do
7 tail -n +2 “$file” >> $new_file
8 done

1 – Name of the file you are creating
2 – 4 Check if the file already exists.  If so, then remove it
5 Loop through the smaller files.  Note that I have a ? to make sure that it is not trying to add the big file back to itself.
6-8 Concatenate the file to the $new_file, but ONLY lines 2-end of file.  This omits the header record of the .csv file.

Leave a Reply