Oasist Blog

Deliver posts regarding linguistics, engineering and life at my will.

Automation of Botpress Accuracy Inspection Vol.3 - Rails Application -

f:id:oasist:20201024112839p:plain
Botpress

Contents

1. Introduction

I previously introduced Python and Ruby scripts in Automation of Botpress Accuracy Inspection Vol.1 - CSV → JSON Converter - and Automation of Botpress Accuracy Inspection Vol.2 - Q&A Confidence Matric Chart -.
The other day, I created a Ruby on Rails(simply called Rails below) application to execute them on the web.

To begin with, this application does not access database and some other features powered by the latest Rails, so I start the application with the following options.

$ rails new botpress_inspection_tool_kit_rails -d -M -O -C -T

To know what each option means, please refer to it with rails new --help.


To cut a long story short, this application is quite thin VC model, not MVC.
That is why some implementation is out of range of The Rails Way.
I am not happy with it myself, so I would like to replace this application with a thin framework.

In this article, I will feature its implementation, so please see the manual documents to know how to use it.

2. Prerequisites

  1. You have already built Botpress server and done training of learning data(if not yet, please refer to Ready Botpress).
  2. You have "Bot ID", "User ID" and "Bearer Token"(if not yet, please refer to Prepare for Required Parameters).
  3. You can access the Rails application, Botpress Inspection Tool Kit via a browser.

3. Routing

I did not make use of resource or resources methods because the use case did not apply to CRUD.
The last 2 definitions of routing are for avoidance of GET request to the pages requested by POST method and redirection to the previous page.

config/routes.rb

Rails.application.routes.draw do
  root 'top#index'
  get  '/top'                          => 'top#index'
  get  '/json-converters/select-csv'   => 'json_converters#select_csv'
  post '/json-converters/download'     => 'json_converters#download'
  get  '/converse-api/select-data'     => 'converse_api#select_data'
  post '/converse-api/generate-matrix' => 'converse_api#generate_matrix'
  post '/converse-api/export-matrix'   => 'converse_api#export_matrix', default: { format: :csv }

  get '/json-converters/download'     => redirect('/json-converters/select-csv')
  get '/converse-api/generate-matrix' => redirect('/converse-api/select-data')
end
                      Prefix Verb URI Pattern                             Controller#Action
                        root GET  /                                       top#index
                         top GET  /top(.:format)                          top#index
  json_converters_select_csv GET  /json-converters/select-csv(.:format)   json_converters#select_csv
    json_converters_download POST /json-converters/download(.:format)     json_converters#download
    converse_api_select_data GET  /converse-api/select-data(.:format)     converse_api#select_data
converse_api_generate_matrix POST /converse-api/generate-matrix(.:format) converse_api#generate_matrix
  converse_api_export_matrix POST /converse-api/export-matrix(.:format)   converse_api#export_matrix {:default=>{:format=>:csv}}
                             GET  /json-converters/download(.:format)     redirect(301, /json-converters/select-csv)
                             GET  /converse-api/generate-matrix(.:format) redirect(301, /converse-api/select-data)

4. JSON Converter

4-1. Controller

JsonConvertersController#select_csv is for the form to send a CSV learning data.

JsonConvertersController#download convert it to a JSON format optimized for Boptress Q&A import.
If no file is selected, it shows an error message "Choose a learning data csv."

app/controllers/json_converters_controller.rb

class JsonConvertersController < ApplicationController
  include JsonGenerator

  def select_csv
  end

  def download
    if csv_learning_data = file_params[:csv_learning_data]
      json_learning_data = generate_json_file(csv_learning_data)
      send_data(
        json_learning_data,
        filename: "learning_data_#{DateTime.current.strftime('%F%T').gsub('-', '').gsub(':', '')}.json"
      )
    else
      flash[:alert] = 'Choose a learning data csv.'
      render :select_csv
    end
  end

  private

  def file_params
    params.permit(:csv_learning_data)
  end
end

JsonGenerator#gen_hash_template generates a template hash object.

JsonGenerator#generate_json_file assigns serial numbers to id key, add questions to questions > ja key and a answer to answers > ja.
After it removes duplicate values, it converts the hash object to JSON format.

app/controllers/concerns/json_generator.rb

module JsonGenerator
  def gen_hash_template
    {
      id: '',
      data: {
        action: 'text',
        contexts: [
          'hoge'
        ],
        enabled: true,
        answers: {
          ja: []
        },
        questions: {
          ja: []
        },
        'redirectFlow': '',
        'redirectNode': ''
      }
    }
  end

  def generate_json_file(csv_learning_data)
    learning_data = []
    hash_template = gen_hash_template
    CSV.foreach(csv_learning_data, headers: true) do |learning_datum|
      if hash_template[:data][:answers][:ja].last == learning_datum['Answers']
        hash_template[:data][:questions][:ja] << learning_datum['Questions']
      else
        hash_template = gen_hash_template
        hash_template[:id] = learning_datum['Serial_Nums']
        hash_template[:data][:questions][:ja] << learning_datum['Questions']
        hash_template[:data][:answers][:ja] << learning_datum['Answers']
      end
      hash_template[:data][:questions][:ja].uniq!
      learning_data << hash_template
    end
    JSON.dump({ qnas: learning_data.uniq })
  end
end

4-2. View

A CSV learning data is sent and processed by JsonConvertersController#download action.
To click "Export JSON", a JSON learning data will be downloaded.

<h1>Learning Data Converter from CSV to JSON</h1>
<%= form_with url: json_converters_download_path, method: :post, multipart: true do |f| %>
  <div class="container">
    <p><%= f.label :csv_learning_data, 'Choose CSV Learning Data' %></p>
    <p><%= f.file_field :csv_learning_data, accept: '.csv' %></p>
    <p><%= f.submit 'Export JSON', class: 'btn btn-primary' %></p>
  </div>
<% end %>

5. Converse API

5-1. Controller

ConverseApiController#select_data is for the form to send required parameters and a CSV test data.

ConverseApiController#generate_matrix convert the CSV test data to a CSV format of confidence matrix chart and render HTML.
If any parameter is lacking, it shows an error message "Fill in values or choose files in each field." by the callback function alert_lacking_form_params.

ConverseApiController#export_matrix provides a function to download a CSV format of confidence matrix chart.

As you may find it out, a class variable @@csv_data is used to have the value in common between ConverseApiController#generate_matrix and ConverseApiController#export_matrix.
Basically speaking, class variables must be avoided because they can be read or overwritten anywhere in the class, only to embed unexpected bugs.
Even if I replace it with a call back function, I will not be going to duplicate process, so I chose the easiest way and committed a taboo to implement it with "VC" model.

To excuse myself, that was the last thing I would like to do.

app/controllers/converse_api_controller.rb

class ConverseApiController < ApplicationController
  include ApiCaller
  include MatrixGenerator

  before_action :alert_lacking_form_params, only: %i(generate_matrix)

  def select_data
  end

  def generate_matrix
    url, req = authenticate(
      converse_api_params[:protocol],
      converse_api_params[:host],
      converse_api_params[:bot_id],
      converse_api_params[:user_id],
      converse_api_params[:bearer_token]
    )
    @@csv_data = CSV.generate do |csv|
      test_data = CSV.read(converse_api_params[:csv_test_data], headers: true)
      csv << set_header(test_data['Serial_Nums'])
      answers_arr = test_data['Answers']
      test_data.each do |test_datum|
        begin
          res = get_api_response(test_datum['Questions'], url, req)
        rescue SocketError
          flash[:alert] = 'It failed to successfully create a matrix chart. Input correct Host.'
          return
        end
        begin
          csv << set_row(test_datum, answers_arr, res.body)
        rescue NoMethodError
          flash[:alert] = 'It failed to successfully create a matrix chart. Input correct Bot ID, User ID and Bearer Token.'
          return
        end
      end
    end
    @matrix = @@csv_data.split("\n").map { |str| str.split(',') }
  end

  def export_matrix
    send_data(
      @@csv_data,
      filename: "matrix_#{DateTime.current.strftime('%F%T').gsub('-', '').gsub(':', '')}.csv"
    )
  end

  private

  def converse_api_params
    params.permit(
      :protocol,
      :host,
      :bot_id,
      :user_id,
      :bearer_token,
      :csv_test_data
    )
  end

  def alert_lacking_form_params
    unless converse_api_params[:protocol] && \
          converse_api_params[:host] && \
          converse_api_params[:bot_id] && \
          converse_api_params[:user_id] && \
          converse_api_params[:bearer_token] && \
          converse_api_params[:csv_test_data]
      flash[:alert] = 'Fill in values or choose files in each field.'
      render :select_data
      return
    end
  end
end

app/controllers/concerns/api_caller.rb

ApiCaller#authenticate returns a URL and request with authorization.

get_api_response receives them and a question and call the Converse API.
If the protocol is https, it executes it with SSL.

module ApiCaller
  def authenticate(protocol, host, bot_id, user_id, bearer_token)
    url = URI.parse("#{protocol}://#{host}/api/v1/bots/#{bot_id}/converse/#{user_id}/secured?include=state,suggestions")
    req = Net::HTTP::Post.new(url)
    req[:authorization] = bearer_token
    return url, req
  end

  def get_api_response(question, url, req)
    req.set_form_data(type: :text, text: question)
    net_http = Net::HTTP.new(url.host, url.port)
    net_http.use_ssl = true if url.to_s.include?('https')
    res = net_http.start { |http| http.request(req) }
  end
end

MatrixGenerator#set_header literally sets the header of a matrix chart with ["Serial_Nums", "Test_Data"] combined with serial numbers in the CSV test data.

MatrixGenerator#set_answers_confidence create a hash object of confidence, { question: confidence }.

MatrixGenerator#set_row literally sets the rows of a matrix chart with [{serial number}, {test_question}] combined with confidence values provided by MatrixGenerator#set_answers_confidence.

app/controllers/concerns/matrix_generator.rb

module MatrixGenerator
  def set_header(serial_nums)
    header = %w(Serial_Nums Test_Data)
    header.concat(serial_nums)
  end

  def set_answers_confidence(res_body)
    res_hash = JSON.parse(res_body)
    answers_confidence = 0.upto(res_hash.dig('suggestions').size - 1).map do |i|
      { res_hash.dig('suggestions')[i].dig('payloads')[1].dig('text') => res_hash.dig('suggestions')[i].dig('confidence') }
    end
  end

  def set_row(test_datum, answers_arr, res_body)
    row = [test_datum['Serial_Nums'], test_datum['Questions']]
    last_index = answers_arr.size - 1
    confidence = [].fill('0.0%', 0..last_index)
    answers_confidence = set_answers_confidence(res_body)
    answers_confidence.each do |ans_conf|
      index = answers_arr.find_index(ans_conf.keys.first)
      confidence[index] = "#{sprintf('%.1f', ans_conf.values.first * 100)}%"
    end
    row.concat(confidence)
  end
end

5-2. View

Each value and CSV test data are sent and processed by ConverseApiController#export_matrix action.
To click "Generate Matrix Chart", a matrix chart of confidence will show up.

app/views/converse_api/select_data.html.erb

<h1>Generate Matrix Chart of Confidence</h1>
<%= form_with url: converse_api_generate_matrix_path, method: :post, multipart: true do |f| %>
  <div class="form-item">
    <p><strong><%= f.label :protocol, 'Protocol' %></strong></p>
    <p><%= f.select :protocol, [['http'], ['https']], { selected: 'http' } %></p>
  </div>
  <div class="form-item">
    <p><strong><%= f.label :host, 'Host' %></strong></p>
    <p class="note">* Port is required in the local environment</p>
    <p><%= f.text_field :host %></p>
  </div>
  <div class="form-item">
    <p><strong><%= f.label :bot_id, 'Bot ID' %></strong></p>
    <p><%= f.text_field :bot_id %></p>
  </div>
  <div class="form-item">
    <p><strong><%= f.label :user_id, 'User ID' %></strong></p>
    <p><%= f.text_field :user_id %></p>
  </div>
  <div class="form-item">
    <p><strong><%= f.label :bearer_token, 'Bearer Token' %></strong></p>
    <p><%= f.text_area :bearer_token, size: '100x5' %></p>
  </div>
  <div class="form-item">
    <p><strong><%= f.label :csv_test_data, 'Choose Test Data CSV' %></strong></p>
    <p><%= f.file_field :csv_test_data, accept: '.csv' %></p>
  </div>
  <div class="form-item">
    <p><%= f.submit 'Generate Matrix Chart', class: 'btn btn-primary' %></p>
  </div>
<% end %

The matrix chart is rendered row by row.
Its confidence is evaluated by ConverseApiHelper#evaluate_confidence helper method, and it paints the cell a specific colour according to the value.

app/views/converse_api/generate_matrix.html.erb

<h1>Matrix Chart of Confidence</h1>
<p><%= link_to 'Export CSV', converse_api_export_matrix_path, method: :post, class: 'btn btn-primary' %></p>
<div class="scroll-table">
  <table>
    <% if @matrix %>
      <% @matrix.each do |matrix| %>
        <tr>
          <% matrix.each do |row| %>
            <th class=<%= evaluate_confidence(row) %>><%= row %></th>
          <% end %>
        </tr>
      <% end %>
    <% end %>
  </table>
</div>

<div class="legend">
  <h2>Legend</h2>
  <p class="excellent">Greater than or Equal to 70.0%</p>
  <p class="good">Greater than or Equal to 50.0% and less than 70.0%</p>
  <p class="bad">Greater than or Equal to 30.0% and less than 50.0%</p>
  <p class="useless">Greater than or Equal to 0.1% and less than 30.0%</p>
</div>

  • Greater than or Equal to 70.0%: .excellent
  • Greater than or Equal to 50.0% and less than 70.0%: .good
  • Greater than or Equal to 30.0% and less than 50.0%: .bad
  • Greater than or Equal to 0.1% and less than 30.0%: .useless

app/helpers/converse_api_helper.rb

module ConverseApiHelper
  def evaluate_confidence(row)
    return unless row.include?('%')
    if row.to_f == 0.0
      ''
    elsif row.to_f >= 70.0
      'excellent'
    elsif row.to_f >= 50.0
      'good'
    elsif row.to_f >= 30.0
      'bad'
    else
      'useless'
    end
  end
end

app/assets/stylesheets/application.css

.excellent {
  background-color: #1971ff;
  color: #fff;
}

.good {
  background-color: #00b06b;
}

.bad {
  background-color: #f2e700;
}

.useless {
  background-color: #ff4b00;
  color: #fff;
}

.legend {
  max-width: 40%;
  margin-top: 30px;
  padding: 15px;
  background-color: #ededed;
}

.legend h2 {
  margin-top: 0;
  margin-bottom: 10px;
}

.legend p {
  margin: 0;
  padding: 5px;
}

6. E2E Test

To use System Spec, add the following settings.

spec/rails_helper.rb

# This file is copied to spec/ when you run 'rails generate rspec:install'
require 'spec_helper'
ENV['RAILS_ENV'] ||= 'test'

require File.expand_path('../config/environment', __dir__)

RSpec.configure do |config|
  config.include DownloadHelper, type: :system, js: true
  config.before(:suite) { Dir.mkdir(DownloadHelper::PATH) unless Dir.exist?(DownloadHelper::PATH) }
  config.after(:example, type: :system, js: true) { clear_downloads }
...
  config.before(:each) do |example|
    if example.metadata[:type] == :system
      driven_by(:selenium, using: :headless_chrome, screen_size: [1400, 1080]) do |option|
        option.add_argument('no-sandbox')
        option.add_argument('--lang=ja-jp')
      end

      page.driver.browser.download_path = DownloadHelper::PATH
    end
  end
end

This helper provides methods to simulate file downloads.

spec/support/download.rb

module DownloadHelper
  TIMEOUT = 10
  PATH    = Rails.root.join("tmp/downloads")

  extend self

  def downloads
    Dir[PATH.join("*")]
  end

  def download
    downloads.last
  end

  def download_content
    wait_for_download
    File.read(download)
  end

  def wait_for_download
    Timeout.timeout(TIMEOUT) do
      sleep 0.1 until downloaded?
    end
  end

  def downloaded?
    !downloading? && downloads.any?
  end

  def downloading?
    downloads.grep(/\.crdownload$/).any?
  end

  def clear_downloads
    FileUtils.rm_f(downloads)
  end

  def download_file_name
    wait_for_download
    File.basename(download)
  end
end

spec/system/json_converters_spec.rb

require 'rails_helper'

RSpec.describe "ConverseApi", type: :system do
  before do
    visit json_converters_select_csv_path
  end

  describe 'Convert learning CSV data to JSON data' do
    it 'enables users to get to json_converters_select_csv_path' do
      expect(page).to have_current_path json_converters_select_csv_path
    end

    context 'CSV file is chosen' do
      it 'succeeds in converting CSV file to JSON file' do
        attach_file 'Choose CSV Learning Data', "#{Rails.root}/spec/factories/learning_data.csv"
        click_on 'Export JSON'
        expect(download_file_name).to match(/learning_data.*json/)
      end
    end

    context 'CSV file is NOT chosen' do
      it 'shows error message' do
        click_on 'Export JSON'
        expect(page).to have_selector '.alert-danger', text: 'Choose a learning data csv.'
      end
    end
  end
end

spec/system/converse_api_spec.rb

require 'rails_helper'

RSpec.describe "ConverseApi", type: :system do
  before do
    visit converse_api_select_data_path
  end

  describe 'Render HTML matrix chart and doanload CSV' do
    it 'enables users to get to converse_api_select_data_path' do
      expect(page).to have_current_path converse_api_select_data_path
    end

    context 'Form items are filled with proper values' do
      before do
        select 'https', from: 'protocol'
        fill_in 'Host', with: ENV['BOTPRESS_HOST']
        fill_in 'Bot ID', with: ENV['BOTPRESS_BOT_ID']
        fill_in 'User ID', with: ENV['BOTPRESS_USER_ID']
        fill_in 'Bearer Token', with: ENV['BOTPRESS_BEARER']
        attach_file 'Choose Test Data CSV', "#{Rails.root}/spec/factories/test_data.csv"
        click_on 'Generate Matrix Chart'
      end

      it 'succeeds in rendering HTML matrix chart' do
        expect(page).to have_current_path converse_api_generate_matrix_path
      end

      it 'succeeds in downloading CSV matrix chart' do
        click_on 'Export CSV'
        expect(download_file_name).to match(/matrix.*csv/)
      end

      it 'returns to root_path when page is reloaded' do
        visit current_path
        expect(page).to have_current_path converse_api_select_data_path
      end
    end

    context 'Form items are filled with random Host' do
      before do
        select 'https', from: 'protocol'
        fill_in 'Host', with: 'foo'
        fill_in 'Bot ID', with: ENV['BOTPRESS_BOT_ID']
        fill_in 'User ID', with: ENV['BOTPRESS_USER_ID']
        fill_in 'Bearer Token', with: ENV['BOTPRESS_BEARER']
        attach_file 'Choose Test Data CSV', "#{Rails.root}/spec/factories/test_data.csv"
        click_on 'Generate Matrix Chart'
      end

      it 'fails in rendering HTML matrix chart and an error message is shown' do
        expect(page).to have_current_path converse_api_generate_matrix_path
        expect(page).to have_selector '.alert-danger', text: 'It failed to successfully create a matrix chart. Input correct Host.'
      end
    end

    context 'Form items are filled with random BotID, UserID and Bearer Token' do
      before do
        select 'https', from: 'protocol'
        fill_in 'Host', with: ENV['BOTPRESS_HOST']
        fill_in 'Bot ID', with: 'foo'
        fill_in 'User ID', with: 'bar'
        fill_in 'Bearer Token', with: 'piyo'
        attach_file 'Choose Test Data CSV', "#{Rails.root}/spec/factories/test_data.csv"
        click_on 'Generate Matrix Chart'
      end

      it 'fails in rendering HTML matrix chart and an error message is shown' do
        expect(page).to have_current_path converse_api_generate_matrix_path
        expect(page).to have_selector '.alert-danger', text: 'It failed to successfully create a matrix chart. Input correct Bot ID, User ID and Bearer Token.'
      end
    end

    context 'CSV file is NOT chosen' do
      it 'succeeds in converting CSV file to JSON file' do
        click_on 'Generate Matrix Chart'
        expect(page).to have_selector '.alert-danger', text: 'Fill in values or choose files in each field.'
      end
    end
  end
end

7. Conclusion

Rails is too rich to implement a "VC" model application, and I feel like it is foolish that I got rid of many useful features to make the application as thin as possible.
I felt it difficult to code without model layers, so it was an weird experience.

I am not definitely happy with this implementation, so I would like to replace it with a thin framework.

8. Deliverables