In early 2021 I was traveling from San Francisco, CA to Phoenix, AZ and I noticed an annoying phenomenon wherein depending on which gas station you go to the price would vary by as much as $1.20 per gallon. At one point I saw an attractive diesel price, but only upon exiting the I-5 freeway did the signage for the much higher regular gasoline price present itself. Thanks to the wonders of Google Maps I was able to take a few moments to plot my course towards a reasonably priced Flying J branch, but the experience planted the seed in my head to create this project.
I wanted a GIS that would scrape the internet for current gas prices from anyplace willing to provide them and plot them out in an easy to use interface. For my initial proof-of-concept I settled on the following three sources of data; Pilot/Flying J Truck Stops, Loves Truck Stops, and Safeway Fuel.
Pilot/Flying J Truck Stops:
This company provides convenient CSV files providing the current pricing for all of their locations. In a separate CSV file they provide the Address, GPS Coordinates, and amenities of each location. With some finessing I was able to produce a usable primary-key foreign-key pair between the two files and extract all of the data I need.
Loves Truck Stops:
This company provides a Microsoft Excel spreadsheet that you can click a button on their website to download. I’m still working on a clean solution to automate the fetching of this file, I have Selenium on my radar but would prefer to find a more direct method. Using the openpyxl library it was very straightforward to extract all of the needed data from this Excel file. Loves was nice enough to provide the pricing and GPS coordinates for all of their locations in the single spreadsheet.
This company does not provide any sort of spreadsheet or API, to get the gas prices you would have to browse the ~280 pages for each individual branch and read it off. So we browse the ~280 pages for each individual branch and read it off. My short Bash script iterates through the list of URLs and uses the links web browser to fetch the page contents in plaintext format. Some Unix-philosophy faithful piping produces a list of URLs with associated price, and joining this with my pre-made table of Geocoded locations provides us with all of the attributes we need.