Automated Big Data Processing Control Box

In this project we process half a peta byte (.5PB) of raw data. Initially, we fuse multiple reference ground truth sheets of log data using JMP software. Then we use robo copy (bash script for robust data copying) invoked by python script which first allocats the target external hard drive to copy data over and automatically look up target log data to copy. We build a complete data copy then process pipeline that after data is being copied to target external drives, a set of python scripts process these logs by first triming the target log files to certain log time periods via generating a csv file with target start and end times for each respective log. After that, another python script extract meta data from the trimmed raw logs and thereafter, another script handles extracting the target data (stereo images) from the raw trimmed files. The following diagram shows the process of preporcessing data files, copying data files, and post processing raw data. a traditional control box to control operation of 5HP AC motor. Safey measures are ensured by using an RSTP controller.
Technologies Used: Contactor, Mechanical protection, RSTP, Overloads.