{"id":13921,"date":"2016-07-14T12:15:27","date_gmt":"2016-07-14T09:15:27","guid":{"rendered":"https:\/\/www.webcodegeeks.com\/?p=13921"},"modified":"2016-07-13T17:03:55","modified_gmt":"2016-07-13T14:03:55","slug":"capybara-selenium-testing-scraping","status":"publish","type":"post","link":"https:\/\/www.webcodegeeks.com\/web-development\/capybara-selenium-testing-scraping\/","title":{"rendered":"Capybara and Selenium for Testing and Scraping"},"content":{"rendered":"

Capybara<\/a>, aside from being the largest rodent in the world<\/a>, is also a fantastic tool to aid you in interacting with browser functionality in your code, either for testing or just to interact with or scrape data from a website.<\/p>\n

Capybara isn\u2019t what actually interacts with the website \u2014 rather, it\u2019s a layer that sits between you and the actual web driver. This could be Selenium, PhantomJS, or any of the other drivers that Capybara supports. It provides a common interface and a large number of helper methods for extracting information, inputting data, testing, or clicking around.<\/p>\n

Just like any abstraction, sometimes you need to go deeper, and Capybara won\u2019t stop you from doing that. You can easily bypass it to get at the underlying drivers if you need more fine-tuned functionality.<\/p>\n

Testing with Capybara<\/h2>\n

Capybara integrates really nicely with all of the common test frameworks used with Rails. It has extensions for RSpec, Cucumber, Test::Unit, and Minitest. It\u2019s used mostly with integration (or feature) tests, which test not so much a single piece of functionality but rather an entire user flow.<\/p>\n

You can use Capybara to test whether certain content exists on the page or to input data into a form and then submit it. This is where you try to ensure that the same key flows (such as registration, checkout, etc.) that your user will take work not just in isolation but flow nicely from one to another.<\/p>\n

With RSpec, we need to first ensure that in our rspec_helper.rb<\/code> file we include the line require 'capybara\/rails'<\/code>. Next, let\u2019s create a new folder called features<\/code> where we\u2019ll put all of the tests which include Capybara.<\/p>\n

Imagine that we have an application for managing coffee farms. In this application, creating a coffee farm is one of the most important functions you can perform, and therefore should be tested thoroughly.<\/p>\n

# spec\/features\/creating_farm_spec.rb\r\nrequire 'rails_helper'\r\n\r\nRSpec.describe 'creating a farm', type: :feature do\r\n  it 'successfully creates farm' do\r\n    visit '\/farms'\r\n    click_link 'New Farm'\r\n\r\n    within '#new_farm' do\r\n      fill_in 'Acres', with: 10\r\n      fill_in 'Name', with: 'Castillo Andino'\r\n      fill_in 'Owner', with: 'Albita'\r\n      fill_in 'Address', with: 'Andes, Colombia'\r\n      fill_in 'Varieties', with: 'Colombia, Geisha, Bourbon'\r\n      fill_in 'Rating', with: 10\r\n    end\r\n    click_button 'Create Farm'\r\n\r\n    expect(page).to have_content 'Farm was successfully created.'\r\n    expect(page).to have_content 'Castillo Andino'\r\n  end\r\nend<\/pre>\n
There are a few things to note with Capybara. The first is that it provides a ton of great helpers such as click_link<\/code>, fill_in<\/code>, click_button<\/code>, etc. Many of these helpers provide a variety of ways to actually find the HTML element that you\u2019re looking for.<\/p>\n
In the example above, we see CSS selectors used with the within<\/code> method. We also see selecting and filling in an input<\/code> field by using the text in its label<\/code>.<\/p>\n
There\u2019s also a third way, not shown here, which allows you to select elements using xpath<\/code>. While xpath<\/code> is the most powerful for selecting, it\u2019s the least clear way. For the purposes of your own sanity, you should probably aim to include an ID<\/code> or class<\/code> property in your HTML to ensure that selecting is straightforward.<\/p>\n
Scraping with Capybara<\/h2>\n
Capybara isn\u2019t only for testing. It can also be used in web scraping. I\u2019ll admit that it isn\u2019t the fastest method, and if all you are doing is visiting a page to extract information without too much interaction with the DOM in terms of data input or clicking, it may not be the best approach. For that, you may want to investigate something like mechanize<\/a> or even nokogiri<\/a> if all you are doing is reading HTML to extract information from it.<\/p>\n
But for the situation where you maybe have to first log in as a user, click on a tab, and then extract some information, this is the sweet spot for Capybara.<\/p>\n
I\u2019ve recently had to rent a car, and I ended up using Hotwire<\/a> for this. Let\u2019s use Capybara to log in and retrieve my confirmation number. In this case, it would be more difficult to use a different scraping tool because it is an Angular SPA, so Capybara works perfectly.<\/p>\n
I\u2019ll create a Rake task which will log in to my account and then loop through all of the confirmation codes and print them to the screen. I\u2019ve used an xpath<\/code> selector here to show that even if there isn\u2019t an easy CSS selector to use, you can still find the element that you\u2019re looking for. This also demonstrates how to use Capybara outside of your testing environment.<\/p>\n
namespace :automate do\r\n  desc 'Grab hotwire confirmation code'\r\n  task hotwire: :environment do |t, args|\r\n    session = Capybara::Session.new(:selenium)\r\n    session.visit 'https:\/\/www.hotwire.com\/checkout\/#!\/account\/login'\r\n\r\n    session.find('#sign-in-email').set(ENV.fetch('EMAIL'))\r\n    session.find(:xpath, '\/\/input[@type=\"password\"]').set(ENV.fetch('PASSWORD'))\r\n    session.find('.hw-btn-primary').click\r\n\r\n    session.all('.confirmation-code').each do |code|\r\n      puts code.text\r\n    end\r\n  end\r\nend<\/pre>\n
On the screen, we get Car confirmation 31233321CA3<\/code> outputted (not my real confirmation number, of course).<\/p>\n
Any time we use the find<\/code> method or all<\/code>, we are given an instance of the Capybara::Node::Element<\/code> object. This object allows us to click<\/code> it, extract the text<\/code>, ask for its DOM path<\/code>, and interact with it in a variety of other ways.<\/p>\n
One other interesting method is the native<\/code> method, which returns us the underlying driver\u2019s object. In this case, it\u2019s an instance of Selenium::WebDriver::Element<\/code> because we are using Selenium. As useful of an abstraction as Capybara is, there will always be times when you need to gain access to the underlying layer.<\/p>\n
As you can see, this could be an easy way to automate a task that has no other alternative than to use the \u201cScreen Scraping\u201d approach. Keep in mind that this is quite brittle, as a slight change to one of their classes or IDs means that the whole thing will stop working.<\/p>\n
Interacting with JavaScript<\/h2>\n
One of the things that Capybara gives you is the ability to interact with your webpages using JavaScript. You aren\u2019t limited to only using Ruby to find and interact with the DOM nodes on your page.<\/p>\n
Capybara gives you two methods to invoke some JavaScript:<\/p>\n