Oasist Blog

This blog features Linguistics, Engineering&Programming and Life Career.

Automation of Botpress Accuracy Inspection Vol.2 - Q&A Confidence Matric Chart -

f:id:oasist:20201024112839p:plain
Botpress

Contents

1. Deliverables

Export CSV which shows confidence to each Q&A.

Serial_Nums Test_Data QA001 QA002 QA003 QA004 QA005 QA006 QA007 QA008 QA009 QA010 QA011 QA012 QA013 QA014 QA015 QA016 QA017 QA018 QA019 QA020 QA021 QA022 QA023 QA024 QA025 QA026 QA027
QA001 GitHubとは何ぞや 57.6% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 42.4% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA002 GitHubに関して 0.0% 75.2% 24.8% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA003 GitHubの登録はどうやるの 0.0% 0.0% 66.7% 33.3% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA004 GitHub登録ができない 0.0% 0.0% 79.2% 20.8% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA005 パスワードを失念した 0.0% 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA006 GitHubを使うのはどうして 0.0% 0.0% 0.0% 0.0% 0.0% 98.8% 0.0% 0.0% 0.0% 0.0% 1.2% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA007 ログインでエラー発生 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 99.3% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.7% 0.0% 0.0% 0.0% 0.0%
QA008 「コンフリクトを解消してください」の表示はどうすればいい 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 60.4% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA009 404エラーが出る 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 4.8% 0.0% 95.2% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA010 GitHubの他との違いは何ぞや 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 3.7% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA011 GitHubはタダで使えるのか 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 4.4% 95.6% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA012 メールアドレス情報の編集 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 67.1% 0.0% 0.0% 32.9% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA013 メールアドレスを変えたんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 14.4% 85.6% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA014 ユーザー名を変えたいんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 88.1% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 11.9% 0.0%
QA015 GitHubアカウントを消したい 0.0% 0.0% 24.3% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 75.7% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA016 メールアドレスを追加するには 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
QA017 SSHキーを登録したんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 73.5% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 26.5% 0.0%
QA018 SSHキーを確認したいんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 78.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 22.0% 0.0%
QA019 Prime Videoとは何ぞや 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 34.0% 0.0% 0.0% 0.0% 0.0% 0.0% 66.0% 0.0% 0.0%
QA020 Prime Videoに関して 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 54.4% 0.0% 0.0% 0.0% 0.0% 45.6% 0.0% 0.0%
QA021 Prime Videoの登録はどうやるの 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 85.7% 0.0% 0.0% 0.0% 14.3% 0.0% 0.0%
QA022 Prime Videoが使えない 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 33.4% 0.0% 0.0% 66.6% 0.0% 0.0%
QA023 Prime Videoでエラー発生 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 99.4% 0.0% 0.6% 0.0% 0.0%
QA024 Prime Videoの他との違いは何ぞや 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 1.0% 0.0% 0.0% 0.0%
QA025 Prime Videoはタダで使えるのか 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 37.0% 0.0% 0.0% 0.0% 63.0% 0.0% 0.0%
QA026 メールアドレスはどこからどうやって変更できるか 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 11.1% 0.0%
QA027 Prime Videoを解約したんだけど 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 19.7% 0.0% 0.0% 0.0% 80.3%

2. Prepare for Required Parameters

2-1. Bearer

Bearer value for authorization is required in the request header.
Access to the top page of Botpress, open inspector, switch to "Network" tab and get Authorization value in "Request Headers" of "users".
Add it to ~/.bash_profile as a environmental variable.

f:id:oasist:20201025192822p:plain
Botpress Bearer

echo 'export BOTPRESS_BEARER="Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImJpbi5saGt0eS5ndHRoai5vLnpzQGdtYWlsLmNvbSIsInN0cmF0ZWd5IjoiZGVmYXVsdCIsImlzU3VwZXJBZG1pbiI6dHJ1ZSwiaWF0IjoxNjAzNjIwODU2LCJleHAiOjE2MDM2NDI0NTYsImF1ZCI6ImNvbGxhYm9yYXRvcnMifQ.R-fshIOFjpm68cJdTuKTq8q2Xe6BdoN4GrYYSEigjxM"' >> ~/.bash_profile

2-2. User ID

Open Emulator in "Q&A", click the gear icon "Configure Settings" and get USER ID.

f:id:oasist:20201025195113p:plain
User ID 1
f:id:oasist:20201025195226p:plain
User ID 2

2-3. Bot ID

Open Emulator in "Q&A", input a text and get botid in "Raw JSON" on "Conversation Debugger".

f:id:oasist:20201025195354p:plain
Bot ID

3. Implementation

3-1. Generate Request Header and Body

For the datails of Request Body and API Response, please refer to Botpress > Docs > Channels > Converse API > Usage (Debug API).

Only state and suggestions for include query parameters are required to get confidence to each Q&A.

  • In Python, pass host, port bot_id and user_id from Exec File and generate URL string.
    Assign BOTPRESS_BEARER from the environmental variable to Authorization key in Dict of the request header.
def __init__(self, host, port, bot_id, user_id):
    self.url = "http://{}:{}/api/v1/bots/{}/converse/{}/secured?include=state,suggestions".format(host, port, bot_id, user_id)
    self.headers = {
        "Content-Type": "application/json",
        "Authorization": os.environ["BOTPRESS_BEARER"]
    }
  • In Ruby, pass host, port bot_id and user_id from Exec File and parse URL string.
    Generate a request with the URL and assign BOTPRESS_BEARER from the environmental variable to Authorization key.
    This private method is called when a new instance is created, and returns url and req to instance variables.
def authenticate(host, port, bot_id, user_id)
  url = URI.parse("http://#{host}:#{port}/api/v1/bots/#{bot_id}/converse/#{user_id}/secured?include=state,suggestions")
  req = Net::HTTP::Post.new(url)
  req[:authorization] = ENV['BOTPRESS_BEARER']
  return url, req
end

3-2. Write CSV Header

Even for 3D matrix chart, write date line by line.
Writing header is easier, so there is no problem if we work on it relaxed.

First, make an initial List and Array which includes Serial_Nums and Test_Data.
Second, access Serial_Nums values and create QA{n} List and Array.
Finally, combine the initial List and Array with QA{n} List and Array.

def set_header(serial_nums):
    header = ["Serial_Nums", "Test_Data"]
    serial_nums_list = []
    for serial_num in serial_nums:
        serial_nums_list.append(serial_num)
    header.extend(serial_nums_list)
    return header
def set_header(test_data)
  header = %w(Serial_Nums Test_Data)
  serial_nums = CSV.read(test_data).map(&:first)
  header.concat(serial_nums)
end

3-3. Get Confidence

Assign the following values to each argument, call ConverseAPI and get response.

Argument Value
key1 type
value1 text
key2 text
value2 Test data in csv/test_data.csv
def get_api_response(url, headers, key1, value1, key2, value2):
    data = {
        key1: value1,
        key2: value2
    }
    req = urllib.request.Request(url, json.dumps(data).encode(), headers)
    res = urllib.request.urlopen(req)
    return res
def get_api_response(key1, value1, key2, value2)
  @req.set_form_data(key1 => value1, key2 => value2)
  res = Net::HTTP.new(@url.host, @url.port).start { |http| http.request(@req) }
end

Pass the response body as an argument and parse JSON to Dict and Hash.
Then, generate an List and Array which includes text and confidence keys of each Q&A.

def set_answers_confidence(res_body):
    res_dict = json.loads(res_body)
    answers_confidence = []
    for i in range(0, len(res_dict["suggestions"])):
        answers_confidence.append(
            {
                "text": res_dict["suggestions"][i]["payloads"][1]["text"],
                "confidence": res_dict["suggestions"][i]["confidence"]
            }
        )
    return answers_confidence
def set_answers_confidence(res_body)
  res_hash = JSON.parse(res_body)
  answers_confidence = 0.upto(res_hash.dig('suggestions').size - 1).map do |i|
    { res_hash.dig('suggestions')[i].dig('payloads')[1].dig('text') => res_hash.dig('suggestions')[i].dig('confidence') }
  end
end

3-4. Write CSV Body

First, generate an initial List and Array of qa_num(Serial_Nums) and input(Test_Data). Second, make an non-duplicate answers array, fill it with 0.0% up to the number of the answers and create a condidence List and Array.
Third, load all elements of answers_confidence List and Array one by one and find index with values of test key in the non-duplicate answers array.
Fourth, confidence to the corresponding index will be overridden.
Finally, combine the initial List and Array and the condidence List and Array.

def set_row(test_datum, qnas, res_body):
    row = [test_datum[0], test_datum[1]]
    answers = csv_to_list(qnas, 2)
    answers_list = uniq_list(answers)
    confidence = ["0.0%" * 1 for i in range(len(answers_list))]
    for ans_conf in set_answers_confidence(res_body):
        index = answers_list.index(ans_conf["text"])
        confidence[index] = "{:.1f}%".format(ans_conf["confidence"] * 100)
    row.extend(confidence)
    return row
def set_row(test_datum, qnas, res_body)
  row = [test_datum['Serial_Nums'], test_datum['Test_Data']]
  answers_arr = CSV.read(qnas, headers: true)['Answers'].uniq!
  last_index = answers_arr.size - 1
  confidence = [].fill('0.0%', 0..last_index)
  answers_confidence = set_answers_confidence(res_body)
  answers_confidence.each do |ans_conf|
    index = answers_arr.find_index(ans_conf.keys.first)
    confidence[index] = "#{sprintf('%.1f', ans_conf.values.first * 100)}%"
  end
  row.concat(confidence)
end

Export the matrix chart created by the process so far as CSV file in the assigned file path.

def export_csv(self, response_matrix, test_data, qnas):
    with open(response_matrix, "w") as w:
        writer = csv.writer(w)
        writer.writerow(set_header(csv_to_list(test_data, 0)))
        with open(test_data) as r:
            reader = csv.reader(r)
            next(reader)
            for test_datum in reader:
                writer.writerow(set_row(test_datum, qnas, get_api_response(self.url, self.headers, "type", "text", "text", test_datum[1]).read()))
def export_csv(response_matrix:, test_data:, qnas:)
  CSV.open(response_matrix, 'w') do |csv|
    csv << set_header(test_data)
    i = 1
    CSV.foreach(test_data, headers: true) do |test_datum|
      @res = get_api_response(:type, "text", :text, test_datum['Questions'])
      csv << set_row(test_datum, qnas, @res.body)
      i += 1
    end
  end
end

4. Conclusion

Implementation in the previous article can be valid only in Botpress, and yet that in this article can be generic as long as the code is fixed based on the JSON format to automatically create a matrix chart.

I will update this blog one by one if I find any useful implementation in my tasks next time.

5. Source Code