Today I Learned

29 posts by błażejkosmowski

Skip callbacks in Rails when saving record

This is valid for at least Rails 6. It is also dependent on low-level implementation, but should be rather safe to use.

def without_update_callbacks(record)
    existing_caller = record.method(:_run_update_callbacks)
    record.define_singleton_method(:_run_update_callbacks, ->(&block){ block.call })

    yield
ensure
    record.define_singleton_method(:_run_update_callbacks, existing_caller)
end

Usage

user = User.last

user.name = "Tony"
user.save # runs callbacks

user.name = "Stark"
without_update_callbacks(user) do 
   user.save # does not run callbacks
end

Have you Cmd-C'd your overcommit and lost your changes?

During overcommit running I have interrupted the process and received following message

Interrupt signal received. Stopping hooks...

⚠  Hook run interrupted by user

then

Unable to restore working tree after pre-commit hooks run:
STDOUT:
STDERR:unable to refresh index

To my horror, all my changes were lost! Fortunately, those were kept in stash, so simple git stash pop helped :) More info here

FactoryBot: Constructing objects using Dry::Types

If you face error similar to the one below

Dry::Struct::Error: [YourClassName.new] :some_attribute_name is missing in Hash input

when building objects of the class using Dry::Types and FactoryBot, be advised, that

Although factory_bot is written to work with ActiveRecord out of the box, it can also work with any Ruby class. For maximum compatibility with ActiveRecord, the default initializer builds all instances by calling new on your build class without any arguments. It then calls attribute writer methods to assign all the attribute values. While that works fine for ActiveRecord, it actually doesn’t work for almost any other Ruby class.

The fix is to add following line to your factory definition

initialize_with  { new(attributes) }

ActiveSupport::IncludeWithRange gotcha

Lets see how ruby implements === for ranges.

As documentation say “Returns true if obj is an element of the range, false otherwise”. Let’s try it out.

2.5.1 :001 > (1..10) === 5
 => true

Looks fine… how about if we compare it to another range?

 2.5.1 :001 > (1..10) === (5..15)
 => false

Seems to work properly again. How about if one range is a sub range of the other one.

2.5.1 :004 > (1..10) === (5..6)
 => false

As expected. Those ranges are not equal after all. Or at least (5..6) is not an element that (1..10) holds.

What is surprising, is what happens if we run the same thing in rails console (5.2.0 at the time of writing). Suddenly

[1] pry(main)> (1..10) === (5..6)
=> true

WAT? It now checks if range is included in the original range! Rails do not override === itself though. After looking at what rails adds to range…

[2] pry(main)> (1..10).class.ancestors
=> [ActiveSupport::EachTimeWithZone,
 ActiveSupport::IncludeTimeWithZone,
 ActiveSupport::IncludeWithRange,
 ActiveSupport::RangeWithFormat,
 Range,
 Enumerable,
 ActiveSupport::ToJsonWithActiveSupportEncoder,
 Object,
 RequireAll,
 PP::ObjectMixin,
 Nori::CoreExt::Object,
 ActiveSupport::Dependencies::Loadable,
 JSON::Ext::Generator::GeneratorMethods::Object,
 ActiveSupport::Tryable,
 Kernel,
 BasicObject]

…we have identified this suspicious module ActiveSupport::IncludeWithRange. Its documentation explains everything.

# Extends the default Range#include? to support range comparisons.
#  (1..5).include?(1..5) # => true
#  (1..5).include?(2..3) # => true
#  (1..5).include?(2..6) # => false

Now guess what ruby’s Range#=== uses behind the scenes

              static VALUE
range_eqq(VALUE range, VALUE val)
{
    return rb_funcall(range, rb_intern("include?"), 1, val);
}

Yes… include?. The consequences are… there are consequences ;) The most annoying one is related to rspec.

expect(1..10).to match(5..6) # => true
expect([1..10]).to include(5..6) # => true
expect([1..10]).to match_array([5..6]) # => true

It is not possible to easily compare array of ranges matching on exact begin and end values yet disregarding actual order. Also the match behaviour is really misleading in my opinion. The only matcher we can use safely here is eq, as expect(1..10).to eq(5..6) will fail properly.

How to change stubbed return value with another stub?

Simple - just re-define spy as a result of another stub

valid_token = instance_double(ValidToken)
allow(ValidToken).to receive(:new) { valid_token }
allow(valid_token).to receive(:to_s) { '123' }
allow(valid_token).to receive(:clear!) do
   allow(valid_token).to receive(:to_s) { '456' }
end
valid_token = ValidToken.new
valid_token.to_s # 123
valid_token.clear!
valid_token.to_s # 456

Making enumerator from method that yields values

Original challenge related to AWS SQS QueuePoller

The challenge was to test a static method that yields multiple values but should stop, when some condition is met.

Let’s imagine such method as follows

class Poller
  def self.poll(condition = -> {})
    counter = 0

    while true do
      yield counter += 1
      break if condition.call
    end
  end
end

The problem is with testing such method. We do not only need to test what it yields, but we also need to test and control when it stops. To control when it stops, we need to access the actual block, but to test what it yields, we either need yield_successive_args matcher or we need to fetch consecutive results.

It is possible by aggregating each yielded value and then asserting them altogether, but the resultant code is not nice. The solution would be to make an Enumerator from the poll method and use next to get consecutive results. It is also easy as described in this blog post. The problem is, that we do not want to write code that is only required by our tests.

So the idea is to add creating enumerators when the block is not provided dynamically to the class when testing.

Poller.define_singleton_method(:poll_with_enum) do |*args, &block| 
    return enum_for(:poll) unless block.present?
    poll_without_enum(*args, &block)
end
# alias_method_chain is deprecated
# Poller.singleton_class.alias_method_chain(:poll, :enum)
Poller.singleton_class.alias_method :poll_without_enum, :poll
Poller.singleton_class.alias_method :poll, :poll_with_enum

if we turn this into a helper…

def with_enumerated(subject, method_name)
  begin
    subject.define_singleton_method("#{method_name}_with_enum") do |*args, &block|
      return enum_for(method_name, *args) unless block.present?
      public_send("#{method_name}_without_enum",*args, &block)
    end

    subject.singleton_class.alias_method "#{method_name}_without_enum", method_name
    subject.singleton_class.alias_method method_name, "#{method_name}_with_enum"

    yield
  ensure
    subject.singleton_class.alias_method method_name, "#{method_name}_without_enum"
    subject.singleton_class.remove_method "#{method_name}_with_enum"
  end
end

…then we could leverage it in our tests!

with_enumerated(Poller, :poll) do
  $stop = false
  poller = Poller.poll(condition = -> { $stop == true })

  first_value = poller.next
  expect(first_value).to eq 1

  $stop = true
  second_value = poller.next
  expect(second_value).to eq 2

  expect { poller.next }.to raise_exception(StopIteration)
end

Stubbing responses from AWS services

We have started integration with Amazon SQS recently and did need to write some unit tests related to it. Unfortunately stubbing AWS client library the regular way turned out to be pretty cumbersome and challenging. Fortunately AWS SDK for ruby provides tools that make it pretty comfortable.

# Simple stubbing...
sqs_response_mock = Aws::SQS::Types::ReceiveMessageResult.new
sqs_response_mock.messages << Aws::SQS::Types::Message.new(body: 'abc')
Aws.config[:sqs] = {
    stub_responses: {
        receive_message: sqs_response_mock
    }
}

# ...allows properly polling the queue
poller = Aws::SQS::QueuePoller.new('https://sqs.us-east-2.amazonaws.com/123/some-queue')
poller.poll do |msg|
  puts msg.body
end

# => abc

Documentation can be found here

Handling uniqueness violations in factories

If it happens that your factories’ cross-dependencies result in creating records that violate uniqueness constraint, you can can fix it quick’n’dirty way

FactoryGirl.define do
  factory :day do
    initialize_with {
      Day.where(date: date).first_or_initialize
    }

    sequence(:date) { |n| Date.current + n.days }
  end
end

Ba aware that this is just a hack, and a hack that would add 1 SQL query to each creation of given object. Preferred way is to fix underlying problem. Still, if you do not care about retrieving the actual instance of object from factory, then different strategy can be used, that would mitigate extra query problem.

FactoryGirl.define do
  factory :day do
    to_create do |day|
      day.save!
    rescue ActiveRecord::RecordNotUnique
      Day.find_by(date: day.date)
    end

    sequence(:date) { |n| Date.current + n.days }
  end
end

If for any reason you really need to handle this kind of case, then introducing a custom create strategy (i.e. find_or_create(:day)) might be a way to go.

Mocking database views in Rspec

Sometimes it is a chore to feed database with all data we need for given database view to return results we are after within specs. Fortunately we can utilize temporary tables to mock such database views!

RSpec.configure do |config|
  ActiveRecord::Migration.verbose = false

  config.before(:all, :use_dummy_db_views) do |_example|
    ActiveRecord::Migration.create_table :my_database_view_entries, id: false, force: true do |t|
      t.integer :some_foreign_key      
      t.string :some_name
      t.string :some_code
    end

    MyDatabaseViewModel.table_name = 'my_database_view_entries'
  end

  config.after(:all, :use_dummy_db_views) do |_example|
    MyDatabaseViewModel.table_name = 'orignal_name_of_my_database_view'
    ActiveRecord::Migration.drop_table :my_database_view_entries, force: true    
  end
end

Then it is just a matter of creating simulated view results, i.e. using factories.

# factory
FactoryBot.define do
  factory :my_database_view_model do
    some_foreign_key 10
    some_name 'Iron Man'
    some_code 'IRON'
  end
end

# some spec
it 'some expectation', :use_dummy_db_views do
  create(:my_database_view_model)
  # ...
end

Running heavy specs on CircleCI on demand

We have some specs that we do want to run only once a day (smoke specs that connect to actual live services). To handle this case we have introduced following setup:

.rspec

#...
--exclude-pattern "spec/smoke/*_spec.rb"

.circleci/config.yml

- run:
    name: run specs
    command: |
      if [[ ${RUN_ALL_TESTS} == "true" ]]; then
        bundle exec rspec --format progress --exclude-pattern ""
      else
        bundle exec rspec --format progress
      fi

This way we can control if some specs are excluded or not, using ENV variable. You can then trigger such build on demand or use CircleCI Workflows.

How to run build on CircleCI recurrently

To run CircleCI build nightly (or at any interval we want), we need a couple of things. First we need to get CircleCI token that will allow us to access CircleCI API.

Next we need a script, that will trigger the build. In example below we also provide an extra build parameter RUN_ALL_TESTS that effectively allows us to conditionally run some specs we do not want to run during regular build.

namespace :ci do
  desc 'Runs build on CircleCI'
  task build: :environment do
    command = 'curl -X POST --header "Content-Type: application/json" ' \
      "--data '{\"build_parameters\": {\"RUN_ALL_TESTS\": \"true\"}}' " \
      'https://circleci.com/api/v1.1/project/github/SomeOwner/' \
      "some_project/tree/master\?circle-token\=#{ENV.fetch('CIRCLE_CI_TOKEN')}"

    `#{command}`
  end
end

Last thing is to schedule the build. Simplest way is to use Heroku Scheduler and just configure it to run rake ci:build command.

Edit: No longer valid! For Circle 2.0 we can use workflows for the same effect!

Making factory_bot work with read-only models

In one of our apps we need to ensure that all models are in read-only mode. Still, for testing purposes we need to be able to create instances of such models. Following code makes it possible.

# spec/models/application_record_spec.rb
require 'rails_helper'

RSpec.describe ApplicationRecord do
  it 'ensures all descendants are read-only' do
    Unit = Class.new(ApplicationRecord)

    expect(Unit.new.readonly?).to eq true
    expect { Unit.create! }.to raise_exception(ActiveRecord::ReadOnlyRecord)
  end

  it 'allows creating objects using factories' do
    Unit = Class.new(ApplicationRecord)

    expect { read_only(:unit) }.to change { Unit.count }.by(1)
  end
  
  it 'disallows updating objects' do
    Unit = Class.new(ApplicationRecord)
    unit = read_only(:unit)

    expect { unit.update_attributes!(name: 'New name') }.to \
      raise_exception(ActiveRecord::ReadOnlyRecord)
  end
end

# spec/support/factory_bot.rb
module FactoryBot
  module Strategy
    class ReadOnly <  Create
      def result(evaluation)
        is_readonly = evaluation.object.readonly?
        evaluation.object.define_singleton_method(:readonly?, -> { false })

        super.tap do |object|
          object.define_singleton_method(:readonly?, -> { is_readonly })
        end
      end
    end
  end
end

FactoryBot.register_strategy(:read_only, FactoryBot::Strategy::ReadOnly)

RSpec.configure do |config|
  config.include FactoryBot::Syntax::Methods
end

# spec/factories/units_factory.rb
FactoryBot.define do
  factory :unit do
    sequence(:name) { |i| "Unit #{i}" }
  end
end

Configuring capybara-screenshot with Heroku

Due to many problems with our capybara-based automations we execute in Sidekiq on Heroku, we did need some visual feedback of what is going wrong. Unfortunately due to read-only nature of Heroku’s file system we did need to customize capybara-screenshot a bit to achieve this functionality.

# initializers/capybara_screenshot.rb

Capybara::Screenshot.s3_configuration = {
  s3_client_credentials: {
    access_key_id: ENV['DEBUG_BUCKET_S3_KEY'],
    secret_access_key: ENV['DEBUG_BUCKET_S3_SECRET'],
    region: ENV['DEBUG_BUCKET_S3_REGION']
  },
  bucket_name: ENV['DEBUG_BUCKET_S3_BUCKET']
}

# Default available methods use lunchy gem. 
# We do neither need nor want that. 
# Hence introducing simplified version.
Capybara::Screenshot.class_eval do
  def self.save_screenshot
    new_saver(Capybara, Capybara.page, false).save
  end
end

Capybara.save_path = '/tmp' # Writeable directory on heroku

Then, we have decided to rescue and re-raise all exceptions, but also save a screenshot in the process…

#...
  automation.perform
rescue => exception # rubocop:disable Style/RescueStandardError
  Capybara::Screenshot.save_screenshot
  raise exception

Selenium::WebDriver::Error::NoSuchDriverError

We’ve recently experienced some peculiar errors when processing capybara-based automation scripts on Heroku. Most of the time, the error returned did not show anything useful…

Selenium::WebDriver::Error::NoSuchDriverError: no such session

yet for a brief period of time, following error was reported when attempting to access capybara session

Selenium::WebDriver::Error::UnknownError: unknown error: session deleted because of page crash
from tab crashed

Finally, after spotting this comment we’ve reduced chrome window size from 1920,1200 to 1440,900 and the problem is no longer present.

The root reason is unknown, but most likely it is at least partially related to running out of memory (reference). Most of recommendations when using docker in this scenario, was to increase shm-size, by providing --shm-size=2g to docker run. That was not an option for us though…

Hope it helps in case you run into similar situation.

Fetching single file from private repository (+ CI)

We had a situation in which we did need to write an API for an app, but decided to keep it in separate repository and deploy as separate app. This API would use original app’s database in read-only mode. The problem was how to prepare database structure for testing purpose. We’ve decided to use structure.sql from original app, but we did want to keep it in sync somehow.

First thing was to get Github’s personal access token

Then, locally we’ve just altered bin/setup to include following code

require 'dotenv/load'
#...
puts "\n== Importing database structure =="
  Dotenv.load
  system! %{curl -H 'Authorization: token #{ENV.fetch('DEVELOPER_ACCESS_TOKEN')}' -H 'Accept: application/vnd.github.v3.raw' -O -L https://api.github.com/repos/OtherApp/other_app/contents/db/structure.sql}
  system! 'mv structure.sql db/structure.sql'

  puts "\n== Preparing database =="
  system! 'RAILS_ENV=test bin/rails db:drop db:create db:structure:load'

while for CircleCi we did need to add following entry to .circleci/config.yml

      - run:
          name: setup-db
          command: |
            curl \
              --header "Authorization: token ${DEVELOPER_ACCESS_TOKEN}" \
              --header "Accept: application/vnd.github.v3.raw" \
              --remote-name \
              --location https://api.github.com/repos/OtherApp/other_app/contents/db/structure.sql
            mv structure.sql db/structure.sql
            bundle exec rake db:create db:structure:load --trace

Obviously do not forget to ensure correct value for DEVELOPER_ACCESS_TOKEN in your .env and on CircleCI.

What to look for when your factory is missing id

So yesterday evening I’ve spotted a factory that was missing its _factory suffix. Easy peasy, quick rename and off we go… yes.. no… suddenly factories linting failed with following error

FactoryGirl::InvalidFactoryError:
  The following factories are invalid:

  * lockoff - PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
  DETAIL:  Failing row contains (null, null, null, {}, {}, null, null, 2018-06-13 20:20:02.606616, 2018-06-13 20:20:02.606616).
  : INSERT INTO "lockoffs" ("created_at", "updated_at") VALUES ($1, $2) RETURNING "id" (ActiveRecord::StatementInvalid)

null value in column "id" ? What? How? After checking absolutely everything reasonable, I’ve checked column definitions. It turned out, that during selecting changes for structure.sql, somebody forgot to add one important piece to commit, that was effectively preventing automatic generation of consecutive primary key values.

--
-- Name: lockoffs id; Type: DEFAULT; Schema: public; Owner: -
--

ALTER TABLE ONLY lockoffs ALTER COLUMN id SET DEFAULT nextval('lockoffs_id_seq'::regclass);

If you ever face similar issue, do not forget to look for something like that as well…

Joining multiple change expectations in rspec

To look for multiple changes on multiple objects, we could aggregate those changes and verify them en-masse, i.e.

expect { SyncReservations.call }.to \
  change { [property_1, property_2].map{ |p| p.reload.synced_at }.
    to([Time.new(2018, 1, 1, 16, 35), Time.new(2018, 1, 1, 16, 35)])

This will pick partial changes though (if only one value changes), so it is not recommended. Another way is nesting expectations, i.e.

expect {
  expect { SyncReservations.call }.to \
    change { property_1.reload.synced_at }.
      to(Time.new(2018, 1, 1, 16, 35))
}.to change { property_2.reload.synced_at }.
  to(Time.new(2018, 1, 1, 16, 35))

This will work, but you will get only one failure at a time. Still it is useful for joining conflicting expectations (i.e. to with not_to). For non conflicting ones, following syntax is recommended:

# using .and
expect { SyncReservations.call }.to \
  change { property_1.reload.synced_at }.
    to(Time.new(2018, 1, 1, 16, 35)).and \
  change { property_2.reload.synced_at }.
    to(Time.new(2018, 1, 1, 16, 35))

or

# using &
expect { SyncReservations.call }.to \
  change { property_1.reload.synced_at }.
  to(Time.new(2018, 1, 1, 16, 35)) &
  change { property_2.reload.synced_at }.
  to(Time.new(2018, 1, 1, 16, 35))

Properly yielding responses in Rspec

We had some problem with configuring response that should be yielded to block… after a short investigation, here are outcomes

allow(collection).to receive(:each).and_yield([1, 2])

will yield array ([1, 2]) to one block variable

allow(collection).to receive(:each).and_yield(1, 2)

will yield two values (1, 2) to two block variables

allow(collection).to receive(:each).and_yield(1).and_yield(2)

will yield two consecutive values (1, 2) to one block variable, twice

Clicked link and still on the same page (capybara)

When writing some automation i’ve tried to navigate to next page using a link…

click_on '6366'

…but saving and showing screenshot resulted in the same page being displayed. The reason was simple, link’s target was a new window (or tab). One way to handle this, is to switch context of window. It is as easy as

switch_to_window(window_opened_by { click_on '6366' })

but if you do not need/want to play around with multiple window contexts, consider following solution

visit find_link('6366')['href']

This way you will stay with current window context :)

Problems matching body params using stub_request?

Surprised that

HTTParty.post(
  'http://example.com/something',
  headers: { content_type: 'application/x-www-form-urlencoded' },
  body: { name: 'Tony', surname: 'Stark'}
)

is not matched by

stub_request(:post, 'http://example.com/something').
  with(
    headers: { 'Content-Type' => 'application/x-www-form-urlencoded' },
    body: { name: 'Tony', surname: 'Stark' }
  ).to_return(body: 'OK')

?

Remember, that HTTP Client does not send actually a hash, and stub_request does not know automagically which encoding you intended to use for your body params. You need to make it explicit, i.e.

stub_request(:post, 'http://example.com/something').
  with(
    headers: { 'Content-Type' => 'application/x-www-form-urlencoded' },
    body: URI.encode_www_form({ name: 'Tony', surname: 'Stark' })
  ).to_return(body: 'OK')

Missing your POST data in controller ?

So you can see JSON payload in inspector…

{days: [{date: "2018-05-18", unitPricing: {1: 285, 2: 285}},…]}

…but cannot see it in controller?

pry(#<Apis::V1::PricingBulkUpdatesController>)> params
=> <ActionController::Parameters {"controller"=>"apis/v1/pricing_bulk_updates", "action"=>"create"} permitted: false>

Remember to POST it with 'Content-Type': 'application/json' header and sell yourself a nice, juicy facepalm :)

Using have_selector with css and exact: option

Using have_selector with css selector and exact: option basically does not work.

I.e.

expect(page).to_not have_content('a.hidden', exact: 'Push pricing to CB')

yields

The :exact option only has an effect on queries using the XPath#is method. Using it with the query "div.hidden" has no effect.

We can use :contains("") css selector though!

expect(page).to_not have_content('a.hidden:contains("Push pricing to CB")')