The ALCHIMIA Project's ETL Pipeline

Fact Sheet This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 101070046. The ALCHIMIA Project's ETL Pipeline Our ETL (Extract, Transform, Load) pipeline is designed to efficiently handle data from two steel plants: FdT and CELSA. This pipeline ensures seamless data extraction, transformation, and storage, facilitating machine learning model training for our partners. Data Source: CELSA Procedure: Data Source: FdT Procedure: Data Collection: Frequency: Data is received from multiple sensors every second. Transport: Sensor data is transmitted to MQTT. 1 2 3 4 Data Transmission: MQTT to Kafka: Data is sent from MQTT to our Kafka topics. Data Storage: Raw Data: Stored in our database from Kafka topics. Data Transformation: Processes: Unit conversions, feature engineering, and other transformations as requested. Tool: Transformations are coded in Python. 5Data Availability: Processed Data Storage: Results are stored in a separate database accessible to end users Data Collection: Method: Data is uploaded to a web service. 1 2 3 4 Data Access: Web Service Client: The data is accessed by our web service client. Data Transmission: Client to Kafka: Data is sent to our Kafka topics. Data Storage and Transformation: Procedure: Follows the same process as Celsa, involving storage in our database, transformation, and availability for machine learning model training. This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 101070046. Technical Stack Programming Language: All code is implemented in Python. Data Streaming and Storage: Kafka: For data topic management. Databases: Separate databases for raw and processed data. Orchestration and Monitoring: Prefect: Used to monitor and deploy our services and manage scheduling (e.g., checking Kafka topics every 10 seconds for new data). This ETL pipeline ensures robust, scalable, and efficient data handling, meeting the needs of our partners for machine learning model training.