Playing around from where I left off last time, the Canadian Securities Administrators (“CSA”) – Disciplined Persons site has loads of information available about disciplined registrants (those registered with securities commissions in one of the 10 provinces and 3 territories within Canada and licensed to sell to and advise Canadians in regards to securities products).

The data goes back a bunch of years, but short of ‘hunting-and-pecking’, name-by-name, the site does not give any overviews on what is going on with ‘Disciplined Persons’. Aggregating the data using a Ruby crawler and “R” to graph the result offers a better view.

Here’s a Treemap showing disciplinary orders issued by various provincial commissions\court:

The next Treemap shows the aggregate data for the penalties meted out by Canadian securities regulators and courts in connection with orders relating securities regulations infractions :

Take special note of 41 disciplined-persons penalized with imprisonment!

Here’s the code to crawl the data:

require 'nokogiri'
require 'rubygems'
require 'watir-webdriver'

browser = Watir::Browser.new :chrome
browser.goto 'https://www.securities-administrators.ca/disciplinedpersons.aspx?id=74'

seen = Array.new

('A'..'Z').each do |letter|

browser.link(:text => letter).click
 browser.table(:id => 'ctl00_bodyContent_gv_list').wait_until_present

loop do
 
 table_Rows = browser.table(:id, 'ctl00_bodyContent_gv_list').rows.length

for i in 1..table_Rows.to_i-1 
 name = browser.table(:id, 'ctl00_bodyContent_gv_list')[i][0].text
 date_of_order = browser.table(:id, 'ctl00_bodyContent_gv_list')[i][1].text
 
 browser.link(:text, name).click
 browser.span(:id => 'ctl00_bodyContent_lbl_persondetail').wait_until_present

discipline_table_Rows = browser.table(:id, 'ctl00_bodyContent_DataList1').rows.length
 for t in 0..discipline_table_Rows.to_i - 1
 
 doc = Nokogiri::HTML.parse(browser.html)
 
 
 t_child = t.to_i + 1
 
 
 if t < 1 && discipline_table_Rows < 2 ### Only one finding ###
 date_of_order = doc.css('#ctl00_bodyContent_DataList1 > tbody > tr > td > table > tbody > tr:nth-child(1) > td').text
 $regulator = browser.table(:id, 'ctl00_bodyContent_DataList1').span(:id, 'ctl00_bodyContent_DataList1_ctl00_lbl_juridiction').div.text
 payment = doc.css('#ctl00_bodyContent_DataList1_ctl00_tr_SanctionList+ tr td').text
 sanctions = doc.css('#ctl00_bodyContent_DataList1_ctl00_lbl_sanction div').text
 violations = doc.css('#ctl00_bodyContent_DataList1_ctl00_lbl_violation div').text
 elsif t < 2 ### More than one finding ###
 date_of_order = doc.css('#ctl00_bodyContent_DataList1 > tbody > tr:nth-child('+t_child.to_s+') > td > table > tbody > tr:nth-child(1) > td').text
 $regulator = browser.table(:id, 'ctl00_bodyContent_DataList1').span(:id, 'ctl00_bodyContent_DataList1_ctl00_lbl_juridiction').div(:class, 'sectiontitle').text
 payment = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_tr_SanctionList+ tr td').text
 sanctions = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_sanction div').text
 violations = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_violation div').text
 else
 date_of_order = doc.css('#ctl00_bodyContent_DataList1 > tbody > tr:nth-child('+t_child.to_s+') > td > table > tbody > tr:nth-child(1) > td').text
 if browser.table(:id 
2pharmaceuticals.com/medicine
   , 'ctl00_bodyContent_DataList1').span(:id, 'ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_juridiction').div(:class, 'sectiontitle').exists?
 $regulator = browser.table(:id, 'ctl00_bodyContent_DataList1').span(:id  , 'ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_juridiction').div(:class, 'sectiontitle').text
 else
 end
 payment = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_tr_SanctionList+ tr td').text
 sanctions = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_sanction div').text
 violations = doc.css('#ctl00_bodyContent_DataList1_ctl0'+t.to_s+'_lbl_violation div').text
 end
 
 
 
 if (seen).include?("#{name}\t#{t}\t#{date_of_order.strip()}\t#{$regulator}\t#{payment.strip()}\t#{sanctions}\t#{violations}")
 
 else
 puts "#{name}\t#{t}\t#{date_of_order.strip()}\t#{$regulator}\t#{payment.strip()}\t#{sanctions}\t#{violations}"
 end
 
 seen << "#{name}\t#{t}\t#{date_of_order.strip()}\t#{$regulator}\t#{payment.strip()}\t#{sanctions}\t#{violations}"
 
 end

 browser.back
 browser.span(:id => 'ctl00_bodyContent_lbl_browse').wait_until_present
 
 end

break if browser.div(:id, 'ctl00_bodyContent_pnl_pager_top').text !~ /\d+\.\.\./
 browser.link(:id, 'ctl00_bodyContent_lbtnNext').click
 sleep (2)
 browser.table(:id => 'ctl00_bodyContent_gv_list').wait_until_present

end

end