pets_seeking_people
Ruby CLI Scraper Gem
I published my first ruby gem! In keeping with the theme of my blog life, I chose something pet-oriented, and built a command line interface that gathers inputs from the user, scrapes a website and supplies the user with a list of dogs or cats available for adoption in their area. In theory, so much fun. I dreamt of all the ascii kitties I could scatter throughout my code. But when I started working on it, I had little time for ascii critters!
There were a couple tough points throughout the project:
-
Determining architecture/file structure: Luckily bundler comes to the rescue a bit on this front. With one command (bundle gem whatever the name of your gem) a scaffold directory is created. There is some placeholder text throughout that needs to be updated with details on the gem, and any additional requirements should be added. But all the essentials are there!
-
Jumping in: Without a test suite, which I could have created….if I hadn’t been distracted by cuddly animals, I was nervous about debugging. So I started small, by mapping out the CLI and using placeholder data to test the methods. Once those looked somewhat normal, I moved to my scraper. Instead of scraping the actual URL to obtain the correct elements, I saved the page’s HTML in a file within my project. That allowed me to test scrape repeatedly without issues.
-
Scraping: Unfortunately, the data I was scraping was in a curious table format. To hone in on the correct elements, I used XPath selectors. The syntax was a little wild at first, but between the Chrome DevTools and some great cheatsheets, the scraper class came together.
-
Actually scraping: The other classes came together rather quickly once the scraper was complete, largely because I stubbed out so much at the beginning. But when the moment of truth came, and I removed my HTML files and scraped the actual website, nothing happened. I made all the emoji faces and I phoned a friend. Using the ChromeDev Tools’ networks tab, we discovered the URLs I was scraping were not the URLs that contained the info I wanted. Fortunately, when I subbed in the request URL I snagged from the networks tab, my scraper scraped away!
Since you’re eager to get started, you can find the gem on ruby gems or find install info on my GitHub repo. Please use, contribute, adopt pets & have all the fun.
Below is a short video about how to use pets_seeking_people. Make sure you catch a glimpse of the ascii puppy!
