As I've been diving deeper into DevOps, I recently built a Weather Data Collection System that combines cloud storage, external APIs, and automated data collection. In this post, I'll walk you through how I created a system that fetches real-time weather data, stores it in AWS S3, and tracks weather conditions across multiple cities. If you're interested in cloud integration and API automation, I hope my experience will help guide your own project!

My Project Goals

When I started this project, I had several key objectives in mind:

• Fetch real-time weather data for multiple cities using the OpenWeather API
• Display key metrics like temperature (°F), humidity, and weather conditions
• Automatically store historical data in AWS S3
• Implement timestamp tracking for weather trend analysis

Through this project, I gained hands-on experience with essential DevOps practices like external API integration, cloud storage management, Infrastructure as Code, and version control.

How I Set Up the Project

I started by cloning my repository:

git clone https://github.com/ShaeInTheCloud/30days-weather-dashboard.git
cd 30days-weather-dashboard

Then I installed my required dependencies using pip:

pip install -r requirements.txt

I needed these main dependencies:

• boto3 for AWS S3 interaction
• python-dotenv for environment variable management
• requests for API calls

My Environment Configuration

I knew security would be crucial when working with APIs and cloud services. I created a .env file in my project root to store sensitive information:

OPENWEATHER_API_KEY=your_api_key
AWS_BUCKET_NAME=your_bucket_name

How I Got My API Keys

1. I created an account on OpenWeather's website
2. I signed up for the free tier API (60 calls/minute limit)
3. I copied my API key to the .env file

My AWS Configuration

I configured my AWS credentials using the AWS CLI:

aws configure

This prompted me for my AWS access key, secret key, and preferred region. I made sure I had appropriate S3 permissions set up in my AWS account.

Running My Application

My main script is located in src/weather_dashboard.py. I run it using:

python src/weather_dashboard.py

The script performs three main functions that I implemented:

1. Connects to OpenWeather API to fetch current weather data
2. Processes and displays the weather metrics
3. Automatically uploads the data to AWS S3 with timestamps

Verifying My Data

After running the script, I check my AWS S3 console to verify the data storage. I can see timestamped weather data files in my specified bucket, making it easy for me to track historical weather patterns.

What I Learned

Building this project taught me several valuable lessons:

1. API Integration: I learned that working with external APIs requires careful error handling and rate limit consideration
2. Cloud Storage: I discovered AWS S3 provides a reliable and scalable solution for storing time-series data
3. Environment Management: I found that keeping sensitive credentials secure through environment variables is crucial
4. DevOps Practices: I realized implementing version control and Infrastructure as Code principles from the start makes the project more maintainable

My Future Plans

I'm planning several enhancements to this project:

1. Implement robust error handling for API and AWS interactions
2. Set up AWS Lambda to automate my data collection on a schedule
3. Create a user interface for my data visualization
4. Add monitoring and alerting for system health
5. Implement data analysis tools for weather pattern recognition

Conclusion

Through building this Weather Data Collection System, I've learned how to combine various DevOps tools and practices into a practical application. I'm excited to continue improving it, and I hope sharing my experience helps others who are starting similar projects.

I want to give a special shout out to Shaeinthecloud for this project idea. Her repo can be found at https://github.com/ShaeInTheCloud/30days-weather-dashboard