I had a unix directory with multiple CSV files, all with the same prefix. I needed them to be concatenated together to make one large file using a shell script.
I used the below code, copied multiple times (one for each file prefix)
Sample files:
file20170804.csv
file20170803.csv
file20170802.csv
new_file=”/dir/file.csv”
if [ -f $new_file ] ; then
rm $new_file
fi
for file in /dir/file?*.csv
do
tail -n +2 “$file” >> $new_file
done
Here is the code with comments:
1 new_file=”/dir/file.csv”
2 if [ -f $new_file ] ; then
3 rm $new_file
4 fi
5 for file in /dir/file?*.csv
6 do
7 tail -n +2 “$file” >> $new_file
8 done
1 – Name of the file you are creating
2 – 4 Check if the file already exists. If so, then remove it
5 Loop through the smaller files. Note that I have a ? to make sure that it is not trying to add the big file back to itself.
6-8 Concatenate the file to the $new_file, but ONLY lines 2-end of file. This omits the header record of the .csv file.