How to Find Historical Weather Data using RapidAPI

Since so many real-world activities depend upon the weather, it is not a surprise that many data science and web-based projects also depend upon accurate and up-to-date weather data to achieve successful results. Data science initiatives such as machine learning and business intelligence get great business insight by comparing business data with historical weather conditions. Internal and consumer-facing websites can benefit from using weather data both to educate users as well as to target specific information or offers based on weather trends and conditions. Taking an even longer view, the changing climate represents both peril and opportunity dependent upon the planning one does around historical weather trends.

Luckily, RapidAPI makes it easy to get historical weather data using the Visual Crossing Weather API. This API is unique because it offers decades of historical weather data, real-time conditions, 15-day forecasts, historical weather summaries, and more for all locations around the globe. However, in this article, we will exclusively focus on how to use it to obtain historical weather reports. We will show how to build a historical weather query, how to execute it in a script using cURL, and how to run it in code such as Python. Finally, we’ll wrap up our discussion by showing how different industries use weather data in their daily operations.

Creating a Weather Query

Building any weather query is easy using the RapidAPI query builder along with the Visual Crossing Weather API documentation for reference. The RapidAPI query builder page shows us the available endpoints as well as a list of the parameters. The Visual Crossing Weather API is easy to use because it fits powerful functionality into only two endpoints, one for historical data and the other for forecasts. In order to run a historical weather query, we will want to select the “Historical weather record” endpoint.

When we select the Historical weather record endpoint, we see all the available parameters along with sample values. Some parameters at the top are pre-populated with values based on our RapidAPI account. Most important among these is the API key. This key is your passport to making queries in RapidAPI.

The “Required Parameters” section is where we will enter the specific details for our weather query. Since each parameter has a detailed description on the RapidAPI page, this article does not need to go explain each option. Instead, we’ll just explain the ones that the user needs to change for a typical query.

startDateTime and its companion endDateTime tell the time range for which we want to find the historical weather. The delta between these two values can be a single hour, multiple days, or even multiple years depending on the data needs.

location provides the specific place for which the query seeks weather data. This can either be a full address such as “1600 Pennsylvania Avenue, Washington DC, USA”, it can be part of an address such as just a city name or a ZIP Code, or it can be a latitude/longitude pair. If an address is given, it will be automatically geocoded before looking up the weather data. That geocoding step will also slow down your weather query slightly. So, if you have latitude and longitude values available, you will want to use them when possible.

Beyond those basic parameters, you can also change the time increment for the returned weather records (aggregateHours), the system of units that you want to use for the output (unitGroup), and the output format that you want (contentType) among others. However, for a basic query, the defaults will provide daily historical data using US standard units with the results being returned as a CSV table.

The RapidAPI GUI will now build a query for us. We simply select our target platform, and a code snippet will be generated automatically. This RapidAPI builder makes querying the weather API very easy.

Running the Weather Query in cURL

Now that our query is defined, we’ll test it using a cURL command line. If you are not already familiar with cURL, it is a very flexible and scriptable command-line tool that can be used to fetch results from nearly any URL-based server. This makes it easy to incorporate cURL into scripts and automation tasks. cURL is common on nearly every Unix and Linux environment and is also installed by default on modern versions of Windows (Windows 10 build 1803 and later, for example).

If you are running an older version of Windows or if you are running on some other OS platform, it is easy to obtain cURL for just about any system. There are many web-based cURL executable downloads available, and one of the most common repositories can be found here. Simply select your OS and follow the appropriate download link.  Of course, the cURL source code is available on GitHub for those who prefer to compile it themselves.

With a single click, RapidAPI will generate the cURL command details for us when we select the “Shell” option and then the “cURL” sub-option in the Code Snippet panel. Here is an example:

curl --request GET 
  --url 'https://visual-crossing-weather.p.rapidapi.com/history?dayStartTime=8%253A00%253A00&contentType=csv&dayEndTime=17%253A00%253A00&shortColumnNames=false&startDateTime=2020-01-01T00%253A00%253A00&aggregateHours=24&location=Herndon%252C%20VA%252C%20USA&endDateTime=2020-01-31T00%253A00%253A00&unitGroup=us' 
  --header 'x-rapidapi-host: visual-crossing-weather.p.rapidapi.com' 
  --header 'x-rapidapi-key: <YOUR_RAPIDAPI_KEY>'

When we look at the structure, we notice three primary parts. After the cURL command itself, we first see the main query URL. If we look inside that URL we see the parameters that were specified on the RapidAPI builder page. The strings are URL encoded so that they can be safely transmitted, and if you decode them you will see that this query sets the date range to 1/1/2020-1/31/2020 and requests daily weather data for “Herndon, VA USA”.

The second two cURL parameters set up the header details. The first is the host component and the second is the RapidAPI key. The generator will automatically set the second one to be your own RapidAPI key. This is important to properly authenticate your account to the system.

On a Linux machine, we can simply paste this query into a command prompt and cURL will fetch a CSV containing the results for us. On Windows, the standard command shell is somewhat less flexible. So, in order to use this command generated by RapidAPI, we will need to make a few modifications. First, edit the command so that it appears on one line and then change the single quotes to double quotes. When you execute the command in cURL, you will get an output like the one shown below ready for use:

Location,Address,Resolved Address,Date time,Maximum Temperature,Minimum Temperature,Temperature,Wind Chill,Heat Index,Precipitation,Snow Depth,Wind Speed,Wind Gust,Cloud Cover,Relative Humidity,Conditions
,"Herndon, VA, United States",,01/01/2020,49.8,27,39.8,23.1,,0,,17.2,21.9,37.1,52.59,Partially cloudy
,"Herndon, VA, United States",,01/02/2020,51,25.1,39.5,21.2,,0,,11.1,,58,62.64,Partially cloudy
,"Herndon, VA, United States",,01/03/2020,51.7,44.9,48.9,40.8,,0.34,,11.4,,89.5,88.41,"Rain, Overcast"
,"Herndon, VA, United States",,01/04/2020,57.6,43,50.8,35.7,,0.49,,23.2,31.1,70.4,85.92,"Rain, Partially cloudy"
,"Herndon, VA, United States",,01/05/2020,43,28.2,36.8,26.5,,0,,24.2,35.7,69.7,62.11,Partially cloudy
…..

Running the Weather Query in Python

With only slightly more effort, we can run the same query in Python. To get RapidAPI to generate the Python code snippet for us, we select Python in the Code Snippets panel. There are current three choices available under Python depending upon your code preferences. The simplest example uses http.client so we’ll select that option from the Python submenu.  The sample code will look like this:

import http.client
 
conn = http.client.HTTPSConnection("visual-crossing-weather.p.rapidapi.com")
 
headers = {
    'x-rapidapi-host': "visual-crossing-weather.p.rapidapi.com",
    'x-rapidapi-key': "<YOUR_RAPIDAPI_KEY>"
    }
 
conn.request("GET", "/history?dayStartTime=8%253A00%253A00&contentType=csv&dayEndTime=17%253A00%253A00&shortColumnNames=false&startDateTime=2020-01-01T00%253A00%253A00&aggregateHours=24&location=Herndon%252C%20VA%252C%20USA&endDateTime=2020-01-31T00%253A00%253A00&unitGroup=us", headers=headers)
 
res = conn.getresponse()
data = res.read()
 
print(data.decode("utf-8"))

After importing the necessary http.client package on the first line, the snippet configures the host for the connection and sets up the headers. As in the cURL example, the RapidAPI key is very important. Without it, the system will not know the account and will refuse to execute the request. Then the code sets up the query URL itself. In this basic case, it uses an HTTP GET request while the main parts of the query string are exactly the same as in the cURL example above. It is easy to see the start and end times as well as the location, although they are URL encoded. Finally, the code submits the request and prints the CSV result to the console.

Running this code snippet is as easy as pasting it into your favorite Python environment. You can paste it into a file on Linux and run it via the comment line (such as “python3 WeatherTest.py”) or even paste the code text directly into your python engine to see the results. The results will be the same table that we obtained when using cURL above.

However, in Python, printing the results in bulk is only the beginning of what we can do with the data. Our next step could be to parse the CSV results with a package such as Python csv. We can then loop through the results, print them however we wish, or store them for later use in the code. We could replace the “print()” command in the code above with something a little more sophisticated such as this:

CSVText = csv.reader(data.decode("utf-8"))
RowIndex = 0

# The first row contain the headers and the additional rows each contain the weather metrics for a single day

# To simply our code, we use the knowledge that column 0 contains the location and column 1 contains the date.  The data starts at column 4

for Row in CSVText:
    if RowIndex == 0:
        FirstRow = Row
    else:
        print('Weather in ', Row[0], ' on ', Row[1])

        ColIndex = 0
        for Col in Row:
            if ColIndex >= 4:
     # Here we can print the weather records, store them, or do whatever we desire
                print('   ', FirstRow[ColIndex], ' = ', Row[ColIndex])
            ColIndex += 1
    RowIndex += 1

What Industries Rely on Weather Data?

We have seen how easy it is to obtain historical weather data using RapidAPI and Visual Crossing Weather. It is also worth surveying how businesses use historical weather data and have made it part of their standard operations. Although nearly all industries benefit from weather analysis, below we’ll showcase a few prime examples.

Historical Weather Data in Retail

Weather data is extremely important to help retail businesses understand their customers’ patterns. While every retailer tracks purchase transactions, forward-thinking retailers also track footfall data and even the paths of customers through the store. This data can be matched against weather data to learn valuable customer patterns. While these patterns sometimes reflect the obvious (rain reduces foot traffic to some types of physical stores and drives traffic to online shops), comparing business data with high-quality historical weather data can identify many subtle patterns that would surprise even a lifelong retailer. For example, one grocer found an unexpected and strong correlation between temperature and bacon sales.

Matching purchase transactions to weather conditions is only the starting point of retail weather analysis, however. If a business tracks data such as customer traffic, footfall, and employee absenteeism, weather data can tell when the store needs to schedule more staff and what conditions are best to turn window shoppers into paying customers. Inventory allocation and management is another key area where weather data adds value. Weather data can be used to refine supply chains, make operations more intelligent, and target weather-triggered ads directly to affected customers using weather alerts.

Historical Weather Data in Energy

The energy industry needs weather data analysis from both a strategic, long-term view and at the specific, individual-customer level. At the strategic level, energy companies need to plan long-term for services and customer needs over time. These decisions include finding the best places to build new assets such as wind farms and understanding the changing climate that will drive long-term customer demand. Historical climate data can help drive optimal decisions on how and where to build energy assets that will not only produce value over time but will also be protected from weather extremes.

Another key component of energy management is at the single building and individual customer level. Usage patterns can be tracked against weather and used to explain energy usage to customers as well as helping customers plan intelligently for their future energy needs. This data can also be used to equip building designers, managers, and individual consumers with the data that they need to improve designs and justify efficiency upgrade decisions.

In addition, energy traders require global, accurate weather and climate data to add insight to their decision making and planning. High-quality weather data helps traders make money by adding speed and intelligence to their trading workflows.

Historical Weather Data in Agriculture

Agriculture is an industry where weather has played a key role since humans first began planting seeds and domesticating livestock. However, modern data science and analysis opened the door to finding insights and gaining efficiencies that were impossible just a few years ago. Automated calculation of growing degree days based on actual, local weather data can help predict exactly when a crop will be ready to harvest. Comparing weather conditions to seed germination rates across various test plots can identify the exact conditions for optimal yield.

Climate change is another area where weather has an outsized effect on agriculture versus other industries. Understanding how the climate has been changing at a given location or in a region offers a valuable road map for future planning. To analyze climate change, however, it takes decades of weather data for the exact field location. Only then can one see how a specific location has changed over decades and better predict what changes the coming years will bring.

Finally, commodity price analysis is another area of historical weather data value in the agriculture industry. Any business that sells, manages, or trades commodities needs a source of weather data for finding historical patterns and predicting optimal times to buy and sell. Accurate historical weather pattern data can make the difference between profit and loss when it comes to managing and trading commodities.

Summary

We’ve only just scratched the surface of explaining how to query historical weather data and how it can be used in various industries. I hope that this article has given you some ideas about new ways to use weather data in your own projects. One key takeaway that should not be lost, however, is how easy it is to get historical weather data using RapidAPI and Visual Crossing Weather. Having easy access to high-quality weather data allows its inclusion in all type of projects from those containing millions of lines of code to simple scripts that an admin can roll together in minutes. It is my hope that this article has not only given you some great ideas for your own projects but has also shown how low the tech barrier is for turning those ideas into action.

Would you like to talk to us about your own weather data ideas? Do you have questions about how to turn your weather use cases into reality? Please reach out to us at info@visualcrossing.com. We love to talk about weather and will be glad to help turn your weather ideas into reality.