Oasist Blog

Deliver posts regarding linguistics, engineering and life at my will.

Automation of Botpress Accuracy Inspection Vol.2 - Q&A Confidence Matric Chart -

f:id:oasist:20201024112839p:plain
Botpress

Contents

1. Deliverables

Export CSV which shows confidence to each Q&A.

Serial_Nums Test_Data QA001 QA002 QA003 QA004 QA005 QA006 QA007 QA008 QA009 QA010 QA011 QA012 QA013 QA014 QA015 QA016 QA017 QA018 QA019 QA020 QA021 QA022 QA023 QA024
QA001 GitHubとは何ぞや 5.3% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA002 GitHubに関して 0.0% 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA003 GitHubの登録の仕方 0.0% 0.0% 69.7% 30.3% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA004 GitHub登録ができない 0.0% 0.0% 60.2% 39.8% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA005 パスワードを失念した 0.0% 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA006 GitHubを使うのはどうして 2.5% 0.0% 0.0% 0.0% 0.0% 97.5% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA007 ログインでエラー発生 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 99.4% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.6% 0.0% 0.0% 0.0%
QA008 「コンフリクトを解消してください」の表示はどうすればいい 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 63.9% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA009 404エラーが出る 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 5.2% 0.0% 94.8% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA010 GitHubの他との違いは何ぞや 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 82.8% 17.2% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA011 GitHubはタダで使えるのか 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 5.6% 94.4% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA012 メールアドレス情報の編集 0.0% 0.0% 0.0% 0.0% 1.3% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 98.7% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA013 ユーザー名を変えたいんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 11.4% 88.6% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA014 GitHubアカウントを消したい 0.0% 0.0% 21.5% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 78.5% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA015 SSHキーを登録したんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 26.2% 0.0% 0.0% 73.8% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA016 SSHキーを確認したいんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 20.3% 0.0% 0.0% 0.0% 79.7% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA017 Prime Videoとは何ぞや 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 4.1% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA018 Prime Videoについて 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA019 Prime Videoの登録はどうやるの 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 87.5% 0.0% 0.0% 0.0% 12.5% 0.0%
QA020 Prime Videoが使えない 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 68.2% 0.0% 0.0% 31.8% 0.0%
QA021 Prime Videoでエラー発生 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 99.4% 0.0% 0.6% 0.0%
QA022 Prime Videoの他との違いは何ぞや 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 12.6% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 87.4% 0.0% 0.0%
QA023 Prime Videoはタダで使えるのか 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 33.1% 0.0% 0.0% 0.0% 66.9% 0.0%
QA024 Prime Videoを解約したいんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 17.9% 0.0% 0.0% 0.0% 0.0% 82.1%

2. Prepare for Required Parameters

2-1. Bearer

Bearer value for authorization is required in the request header.
Access to the top page of Botpress, open inspector, switch to "Network" tab and get Authorization value in "Request Headers" of "users".
Add it to ~/.bash_profile as a environmental variable.

f:id:oasist:20201025192822p:plain
Botpress Bearer

echo 'export BOTPRESS_BEARER="Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImJpbi5saGt0eS5ndHRoai5vLnpzQGdtYWlsLmNvbSIsInN0cmF0ZWd5IjoiZGVmYXVsdCIsImlzU3VwZXJBZG1pbiI6dHJ1ZSwiaWF0IjoxNjAzNjIwODU2LCJleHAiOjE2MDM2NDI0NTYsImF1ZCI6ImNvbGxhYm9yYXRvcnMifQ.R-fshIOFjpm68cJdTuKTq8q2Xe6BdoN4GrYYSEigjxM"' >> ~/.bash_profile

2-2. User ID

Open Emulator in "Q&A", click the gear icon "Configure Settings" and get USER ID.

f:id:oasist:20201025195113p:plain
User ID 1
f:id:oasist:20201025195226p:plain
User ID 2

2-3. Bot ID

Open Emulator in "Q&A", input a text and get botid in "Raw JSON" on "Conversation Debugger".

f:id:oasist:20201025195354p:plain
Bot ID

3. Implementation

3-1. Generate Request Header and Body

For the datails of Request Body and API Response, please refer to Botpress > Docs > Channels > Converse API > Usage (Debug API).

Only state and suggestions for include query parameters are required to get confidence to each Q&A.

  • In Python, pass protocol, host, bot_id and user_id from Exec File and generate URL string.
    Assign BOTPRESS_BEARER from the environmental variable to Authorization key in Dict of the request header.
def __init__(self, protocol, host, bot_id, user_id):
    self.url = "{}://{}/api/v1/bots/{}/converse/{}/secured?include=state,suggestions".format(protocol, host, bot_id, user_id)
    self.headers = {
        "Content-Type": "application/json",
        "Authorization": os.environ["BOTPRESS_BEARER"]
    }
  • In Ruby, pass protocol, host, bot_id and user_id from Exec File and parse URL string.
    Generate a request with the URL and assign BOTPRESS_BEARER from the environmental variable to Authorization key.
    This private method is called when a new instance is created, and returns url and req to instance variables.
def authenticate(protocol, host, bot_id, user_id)
  url = URI.parse("{protocol}://#{host}/api/v1/bots/#{bot_id}/converse/#{user_id}/secured?include=state,suggestions")
  req = Net::HTTP::Post.new(url)
  req[:authorization] = ENV['BOTPRESS_BEARER']
  return url, req
end

3-2. Write CSV Header

Even for 3D matrix chart, write date line by line.
Writing header is easier, so there is no problem if we work on it relaxed.

First, make an initial List and Array which includes Serial_Nums and Test_Data.
Second, access Serial_Nums values and create QA{n} List and Array.
Finally, combine the initial List and Array with QA{n} List and Array.

def set_header(serial_nums):
    header = ["Serial_Nums", "Test_Data"]
    serial_nums_list = data_list(serial_nums)
    header.extend(serial_nums_list)
    return header
def set_header(serial_nums)
  header = %w(Serial_Nums Test_Data)
  header.concat(serial_nums)
end

3-3. Get Confidence

Assign question, url, headers in order to call ConverseAPI and get response.

def get_api_response(question, url, headers):
    data = {
        'type': 'text',
        'text': question
    }
    req = urllib.request.Request(url, json.dumps(data).encode(), headers)
    res = urllib.request.urlopen(req)
    return res
def get_api_response(question, url, req)
  req.set_form_data(type: :text, text: question)
  net_http = Net::HTTP.new(url.host, url.port)
  net_http.use_ssl = true if url.to_s.include?('https')
  res = net_http.start { |http| http.request(req) }
end

Pass the response body as an argument and parse JSON to Dict and Hash.
Then, generate an List and Array which includes text and confidence keys of each Q&A.

def set_answers_confidence(res_body):
    res_dict = json.loads(res_body)
    answers_confidence = []
    for i in range(len(res_dict["suggestions"])):
        answers_confidence.append(
            {
                "text": res_dict["suggestions"][i]["payloads"][1]["text"],
                "confidence": res_dict["suggestions"][i]["confidence"]
            }
        )
    return answers_confidence
def set_answers_confidence(res_body)
  res_hash = JSON.parse(res_body)
  answers_confidence = 0.upto(res_hash.dig('suggestions').size - 1).map do |i|
    { res_hash.dig('suggestions')[i].dig('payloads')[1].dig('text') => res_hash.dig('suggestions')[i].dig('confidence') }
  end
end

3-4. Write CSV Body

First, generate an initial List and Array of qa_num(Serial_Nums) and input(Test_Data). Second, make an non-duplicate answers array, fill it with 0.0% up to the number of the answers and create a confidence List and Array.
Third, load all elements of answers_confidence List and Array one by one and find index with values of test key in the non-duplicate answers array.
Fourth, confidence to the corresponding index will be overridden.
Finally, combine the initial List and Array and the confidence List and Array.

def set_row(test_datum, answers_list, res_body):
    row = [test_datum[0], test_datum[1]]
    confidence = ["0.0%" * 1 for i in range(len(answers_list))]
    for ans_conf in set_answers_confidence(res_body):
        index = answers_list.index(ans_conf["text"])
        confidence[index] = "{:.1f}%".format(ans_conf["confidence"] * 100)
    row.extend(confidence)
    return row
def set_row(test_datum, answers_arr, res_body)
  row = [test_datum['Serial_Nums'], test_datum['Questions']]
  last_index = answers_arr.size - 1
  confidence = [].fill('0.0%', 0..last_index)
  answers_confidence = set_answers_confidence(res_body)
  answers_confidence.each do |ans_conf|
    index = answers_arr.find_index(ans_conf.keys.first)
    confidence[index] = "#{sprintf('%.1f', ans_conf.values.first * 100)}%"
  end
  row.concat(confidence)
end

Export the matrix chart created by the process so far as CSV file in the assigned file path.

def export_csv(self, matrix_chart, test_data):
    with open(matrix_chart, "w") as w:
        writer = csv.writer(w)
        writer.writerow(set_header(csv_to_list_with_index(test_data, 0)))
        with open(test_data) as r:
            reader = csv.reader(r)
            next(reader)
            answers_list = csv_to_list_with_index(test_data, 2)
            for test_datum in reader:
                writer.writerow(set_row(test_datum, answers_list, get_api_response(test_datum[1], self.url, self.headers).read()))
def export_csv(matrix_chart, test_data)
  CSV.open(matrix_chart, 'w') do |csv|
    test_data = CSV.read(test_data, headers: true)
    csv << set_header(test_data['Serial_Nums'])
    answers_arr = test_data['Answers']
    test_data.each do |test_datum|
      @res = get_api_response(test_datum['Questions'], url, req)
      csv << set_row(test_datum, answers_arr, res.body)
    end
  end
end

4. Unit Test

  • TestConverseApi#setUp generates an instance. In Python, it gets response from Converse API with a sample text, where as it gets one by executing ConverseApi#export_csv and access the instance variable res in Ruby.
  • TestConverseApi#test_status_code checks if the API returns 200 in the status code.
  • TestConverseApi#test_has_state_key checks if the response includes state key.
  • TestConverseApi#test_has_suggestions_key checks if the response includes suggestions key.
  • TestConverseApi#test_export_csv checks if the length of CSV test data agrees with that of CSV matrix.

  • Python

import unittest
import json
import csv
import datetime
import sys
sys.path.append("../lib")
sys.path.append("../lib/concerns")
from converse_api import ConverseApi
from api_caller import get_api_response
from file_handler import csv_to_list_with_index

class TestConverseApi(unittest.TestCase):
    def setUp(self):
        protocol          = "https"
        host              = "oasist-botpress-server.herokuapp.com"
        bot_id            = "sample-bot-1"
        user_id           = "oasist"
        self.converse_api = ConverseApi(protocol, host, bot_id, user_id)
        self.res          = get_api_response("GitHubとは", self.converse_api.url, self.converse_api.headers)

    def test_status_code(self):
        self.assertEqual(200, self.res.getcode())

    def test_has_state_key(self):
        self.assertEqual(True, "state" in json.loads(self.res.read()).keys())

    def test_has_suggestions_key(self):
        self.assertEqual(True, "suggestions" in json.loads(self.res.read()).keys())

    def test_export_csv(self):
        matrix_chart  = "../csv/matrix_chart_{0:%Y%m%d}.csv".format(datetime.datetime.now())
        test_data     = "../csv/test_data.csv"
        self.converse_api.export_csv(matrix_chart, test_data)
        test_rows_num   = len(csv_to_list_with_index(test_data, 0))
        matrix_rows_num = len(csv_to_list_with_index(matrix_chart, 0))
        self.assertEqual(test_rows_num, matrix_rows_num)

if __name__ == "__main__":
    unittest.main()
require 'minitest/autorun'
require 'json'
require 'csv'
require_relative '../lib/converse_api'
require_relative '../lib/concerns/matrix_exporter'

class ConverseApiTest < Minitest::Test
  def setup
    protocol      = 'https'
    host          = 'oasist-botpress-server.herokuapp.com'
    bot_id        = 'sample-bot-1'
    user_id       = 'oasist'
    @converse_api = ConverseApi.new(protocol, host, bot_id, user_id)
    @matrix_chart = "../csv/matrix_chart_#{Time.now.strftime("%Y%m%d")}.csv"
    @test_data    = '../csv/test_data.csv'
    @converse_api.export_csv(@matrix_chart, @test_data)
  end

  def test_status_code
    assert_equal '200', @converse_api.res.code
  end

  def test_has_state_key?
    assert_equal true, JSON.parse(@converse_api.res.body).key?('state')
  end

  def test_has_suggestions_key?
    assert_equal true, JSON.parse(@converse_api.res.body).key?('suggestions')
  end

  def test_export_csv
    test_rows_num   = CSV.read(@test_data, headers: true)['Serial_Nums'].size
    matrix_rows_num = CSV.read(@matrix_chart, headers: true)['Serial_Nums'].size
    assert_equal test_rows_num, matrix_rows_num
  end
end

5. Conclusion

Implementation in the previous article can be valid only in Botpress, and yet that in this article can be generic as long as the code is fixed based on the JSON format to automatically create a matrix chart.

I will update this blog one by one if I find any useful implementation in my tasks next time.

6. Source Code