Introduction

Do you lead a busy life? It sure feels like it, but how do you truly know how you spend your time? Would you like to focus your time on specific projects or specific tasks?

The solution is to start tracking you time. I personally, started tracking my time almost five years ago when I had multiple jobs. I was a full-time graduate student with a few part-time job. This created a demand on my time and I needed to manage my time well.

I learnt about Toggl, a free easy to use time tracking application. It is simple to use on android with additional feature on their website. You simply start at timer and save it under a project. You can also capture the task which may be distinct from the project.

Its really up to your own whims on the definition of project. You can track one-off task, like a household repair, but I try to keep these tasks grouped together in a unified project.

I hadn’t really done anything with the data I created tracking my time, so I thought it would be a great opportunity to learn how to create a dashboard in python.

Initialization

As always, we start with loading our python packages. There isn’t really anything out of the ordinary, we will use the requests library to make our API call. You will need to create an account at the Toggl website and create a API key.

I’ve saved the API key in my environmental variables, it is loaded into the script with the os.environ function. You also need to create a header for the API call, this is the part that caused the most difficulty for me as it is very specific in the format.

import pandas as pd
import os
import base64
import requests
import json
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

key = os.environ['Toggl']

string=key+':api_token'
headers={
    'Authorization':'Basic '+base64.b64encode(string.encode('ascii')).decode('utf-8'),
    "Content-Type": "application/json"   
}

Toggl API

This section goes more into the details for the API. The call accepts the following parameters: since for your starting date, until for you finishing date, user_agent this is your user account email and workspace_id which is found on the Toggl website when you click on your workspace. I have kept mine secret, you will need to replace them with your own values.

params = {
    'since':'2020-01-01',
    'until':'2020-12-31',
    'user_agent': 'email',
    'workspace_id' : "workspace"
}

With the header and parameters for the API call setup, we can now use the requests package to make the actual call. The API call returns a json document, for ease I will convert that json into a dataframe. We can than analyze what the API call has returned.

response = requests.get('https://api.track.toggl.com/reports/api/v2/details', headers = headers, params= params)
my_json = json.loads(response.content)
df = pd.DataFrame(my_json['data'])
df['Month'] = pd.DatetimeIndex(df['start']).month
print(df.columns, df.head(5))

Index(['id', 'pid', 'tid', 'uid', 'description', 'start', 'end', 'updated',
       'dur', 'user', 'use_stop', 'client', 'project', 'project_color',
       'project_hex_color', 'task', 'billable', 'is_billable', 'cur', 'tags',
       'Month'],
      dtype='object')            id          pid   tid      uid    description  \
0  1825387165  162435592.0  None  4504525  Data Products   
1  1825095042  162435592.0  None  4504525  Data Products   
2  1824585854  150042491.0  None  4504525        Commute   
3  1824350140  150042504.0  None  4504525   Walmart work   
4  1824350131  150042491.0  None  4504525        Commute   

                       start                        end  \
0  2020-12-31T20:26:49-05:00  2020-12-31T20:44:21-05:00   
1  2020-12-31T16:56:40-05:00  2020-12-31T17:21:47-05:00   
2  2020-12-30T22:55:00-05:00  2020-12-31T23:30:00-05:00   
3  2020-12-30T14:24:04-05:00  2020-12-30T22:55:48-05:00   
4  2020-12-30T14:24:02-05:00  2020-12-30T14:24:04-05:00   

                     updated       dur     user  ...  client  \
0  2021-01-01T20:08:38-05:00   1052000  M2edney  ...    None   
1  2020-12-31T20:26:49-05:00   1507000  M2edney  ...    None   
2  2020-12-31T01:06:18-05:00  88500000  M2edney  ...    None   
3  2020-12-30T22:55:49-05:00  30704000  M2edney  ...    None   
4  2020-12-30T14:24:05-05:00      2000  M2edney  ...    None   

              project project_color project_hex_color  task billable  \
0  Data Science Study             0           #9e5bd9  None     None   
1  Data Science Study             0           #9e5bd9  None     None   
2             Commute             0           #c7af14  None     None   
3             Walmart             0           #d94182  None     None   
4             Commute             0           #c7af14  None     None   

  is_billable   cur tags Month  
0       False  None   []    12  
1       False  None   []    12  
2       False  None   []    12  
3       False  None   []    12  
4       False  None   []    12  

[5 rows x 21 columns]

The issue with this method is that it only returns a limited number of values. We can see it only returned 50 time entries.

print(len(df))

50

To collect all entries within the time frame, we need to try something a little different. The json file does include two important attributes, the total count and the number of entries per page. With these two values, we can loop through all the pages of entries and concatenate them into a single dataframe. The previous attempt already retrieved the first page so we can start at the second. I also included some simple data cleanup to remove null entries and change the duration values into hrs.

page_count = int(my_json['total_count']/my_json['per_page']) + (my_json['total_count'] % my_json['per_page'] > 0)
for page in range(1, page_count):
    params['page']= str(page)
    response = requests.get('https://api.track.toggl.com/reports/api/v2/details', headers = headers, params= params)
    my_json = json.loads(response.content)
    df =pd.concat([df, pd.DataFrame(my_json['data'])])
    
df['dur'] = df['dur']/1000/60/60
df = df[~df['project'].isnull()]
df = df.drop(['tid','updated', 'user', 'use_stop', 'client', 'project_color', 'project_hex_color', 'task', 'billable', 'is_billable', 'cur'],
            axis = 1)
df['Month'] = pd.to_datetime(df['start'], utc=True).dt.month_name()
df= df.sort_values('start')
print(len(df))

863

Simple Graphics

Before creating any complicated graphics for our dashboard, we should make a simple graphics. In this graph we total the duration’s and split them by their labeled project. I use the sns.barplot from theseaborn library because it is well designed and simple to use. This simple graph, shows the total number of hours attributed to each project.

sns.barplot(x='project', y= 'dur', data = df, estimator = 'sum', hue= 'project')
plt.xticks(rotation=45)

([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
 [Text(0, 0, 'Commute'),
  Text(1, 0, 'Thesis work'),
  Text(2, 0, 'Walmart'),
  Text(3, 0, 'Writing course'),
  Text(4, 0, 'CHE215'),
  Text(5, 0, 'CHE615'),
  Text(6, 0, 'Administrative'),
  Text(7, 0, 'CHE616'),
  Text(8, 0, 'CE8201'),
  Text(9, 0, 'Mentoring'),
  Text(10, 0, 'Data Science Study'),
  Text(11, 0, 'Job Seeking')])

Dash Dashboard

The dash dashboard has a pretty simple construction. I start with loading the libraries required. The basic libraries include ‘Dash’ for the basic functions, ‘html’ to design the dashboard layout with html and ‘dcc’ for adding additional core components including radio-buttons and graphs.

Interactive graphs require some additional work. Interactive graphs require the use of the ‘callback’, ‘Input’ and ‘Output’ libraries. The input function is used to define the behaviour of the input componets, sush as the radio button. We assign the button an id and define that the component returned is the ‘value’.

I’ve also taken some monthly average values by using the group_by method. This creates a new dataframe which has multi-indexes which doesn’t work well with plotly express graphs. I drop the indexes to column values with the reset_index method. I finished the summary data by sorting it by the start time with the sort_values method.

# Import packages
from dash import Dash, html, dash_table, dcc, callback, Output, Input, jupyter_dash
import pandas as pd
import plotly.express as px

jupyter_dash.default_mode="external"

# Initialize the app
app = Dash(__name__)

df2= df.groupby(['Month', 'project']).sum().reset_index().sort_values('start')

The next stage of the dashboard is the app.layout. This forms the skeleton of the dashboard and as such utilizes some basic html. Through some experimentation, I have found I’ve liked the styling of the tabs. With the tabs, I can display different content on different tabs. Some tabs will be interactive while others will not.

Tabs are created with the dcc.Tabs function were you provide a list of dcc.Tab functions as children. For any interactive component, you have to provide an “id” so you can reference to it later, this includes the tab setup itself. For this dashboard I’ve decided to have a Datatable on tab one, an interactive annual review chart on tab 2, and a monthly review chart on tab 3. I decided to only make tab 2 interactive as the chart in tab 3 already has some basic interactive features by the nature of being a plotly express graph.

# App layout
app.layout = html.Div([
    dcc.Tabs(id="tabs", value='tab-1', children=[
        dcc.Tab(label='Data Table', value='tab-1'),
        dcc.Tab(label='Annual Review', value='tab-2', children=[
            dcc.RadioItems(options=['avg', 'count', 'sum'], value='sum', id='controls-and-radio-item')]),
        dcc.Tab(label='Monthly Review', value='tab-3'),
    ]),
    html.Div(id='tabs-content')
])

The next stage is the callback functions which runs whenever an input component is changed. Mainly, it is where the interactive inputs and output are declared. You declare them with the previously mention Output and Input functions. It is here were you reference the components based on the ids that you have called them. Your outputs need to be declared prior to your inputs and the order of your inputs will affect your update function. The second part of each declaration is the component that is effected.

@app.callback(
  Output('tabs-content', 'children'),
  Input('tabs', 'value'),
  Input('controls-and-radio-item', 'value')
  )

The final step is to define the effect on your interactive elements in an update function. There are two components here for our dashboard, the updates due to the tab change and the updates from the radio button in the graph on tab 2. With an if statement, we return an HTML Div function/element with the corresponding content for that tab number. For those not familiar with HTML, a Div is just some user defined section.

def render_content(tab, hist):
    if tab == 'tab-1':
        return html.Div([
            dash_table.DataTable(data=df.drop('tags', axis = 1).to_dict('records'), page_size=10)
        ])
    elif tab == 'tab-2':
        return html.Div([
            dcc.Graph(figure = px.histogram(df, x='project', y='dur', color = 'description', histfunc = hist),
                     id='controls-and-graph')   
        ])
    elif tab == 'tab-3':
        return html.Div([
            dcc.Graph(figure = px.line(df2, y = 'dur', x = 'Month', color = 'project', markers=True))          
        ])

# Run the app
if __name__ == '__main__':
    app.run(debug=True)

Dash app running on http://127.0.0.1:8050/

Our program has now run a server to host the dashboard. You view the dashboard by opening the hyperlink or going to the default address ‘http://127.0.0.1:8050/’ in your internet browser. The server is host localy, mean I can not easily display the interactive features.

Conclusion

We have connected to the Toggl service with an API call. With this API call, we collected a years worth of time tracking data to create a local dataframe. Exploring the data with graphs helped us understand the structure and the quality of data in our dataframe. A simple dash dashboard was created with some interactive features takes to the Input and Output functions. This dashboard has multiple tabs, display different types of data on each tab. The first tab simply shows a table of the data as illustrated in the following image:

On the second tab, we’ve created an interactive chart that uses a radio button. This radio button controls how the data in our chart will be aggregated.

And on the final tab, we created a interactive graph with the plotly express library. This chart summaries or data on a month by month basis.

Introduction

This is a continuation from a previous post about creating a custom ChatGPT bot with Langchain. From the previous post, we have created a CSV file which contains the text captured from a folder full of different PDF files. We will continue from there, creating a custom ChatGPT bot from that CSV files.

Loading packages

As with any project, the first step is to load some of the necessary python packages. The load_dotenv function from the dotenv plays a special role in this program. In order to use the Open AI API, we will need to create and Open AI account. With this account, you will receive an API key, which will be connected to any functional calls with your account. You can keep these API keys in your code, but it is better practice to store them in your environmental variables. The environmental variables is a location in windows where you can store some sensitive information that will be only stored on your computer. You can find where your environmental variables are with the windows search, or you can find it under your advanced system properties. Make a new entry in either the user variables or system variables (whichever you prefer) and name it “OPENAI_API_KEY”. The value will be the actual key you get from your Open AI account. After you save your API key, you will need to restart windows. You will need to pay for your API requests for your Open AI account. Using the embedding model turns out to be pretty cheap, while questioning the ChatGPT cost a bit more, usually only a one or two cents per question.

import os
import sys
import pandas as pd
from dotenv import load_dotenv

load_dotenv()

False

Document Loading

The next step is to load your CSV files into a document format used by Lang Chain. I found that the best way to complete this is with the DirectoryLoader and the CSVLoader functions. I have also found that it is important to specify the encoding used to create the CSV files from the pandas dataframes. Furthermore, I’ve also included an additional step which will add 1 to the row numbers in the metadata. The row value represents the page number within the document, but the default indexing for a pandas dataframe starts at 0 and not at 1. Getting the length of our documents represents the overall number of pages that we have scanned from our PDF files.

from langchain.document_loaders import DirectoryLoader, CSVLoader

loader = DirectoryLoader(path='C:/Users/Mark/Desktop/csv_files', glob = '**/*.csv', loader_cls = CSVLoader, use_multithreading = True, show_progress = True, silent_errors = True, loader_kwargs = {'encoding': 'utf-8-sig'})

docs = loader.load()

for doc in docs:
  doc.metadata['row'] +=1
print(len(docs))

  0%|          | 0/3 [00:00<?, ?it/s]100%|██████████| 3/3 [00:00<?, ?it/s]

46

Just to get an idea of the data format, we can print the 10 item stored in our documents.

docs[10]

Document(page_content=": 7\n0: Chatbot - Wikipedia https://en.wikipedia.org/wiki/Chatbot\n\nThe creation and implementation of chatbots is still a developing area, heavily related to artificial intelligence and machine learning,\nso the provided solutions, while possessing obvious advantages, have some important limitations in terms of functionalities and use\ncases. However, this is changing over time.\n\nThe most common limitations are listed below:!83]\n\n= As the input/output database is fixed and limited, chatbots can fail while dealing with an unsaved query.!52]\n\n= A chatbot's efficiency highly depends on language processing and is limited because of irregularities, such as accents and\nmistakes.\n\n= Chatbots are unable to deal with multiple questions at the same time and so conversation opportunities are limited.|83I\n\n= Chatbots require a large amount of conversational data to train. Generative models, which are based on deep learning\nalgorithms to generate new responses word by word based on user input, are usually trained on a large dataset of natural-\nlanguage phrases. |S]\n\n« Chatbots have difficulty managing non-linear conversations that must go back and forth on a topic with a user.|&41\n\n= As it happens usually with technology-led changes in existing services, some consumers, more often than not from older\ngenerations, are uncomfortable with chatbots due to their limited understanding, making it obvious that their requests are being\ndealt with by machines. |83]\n\nChatbots and jobs\n\nChatbots are increasingly present in businesses and often are used to automate tasks that do not require skill-based talents. With\ncustomer service taking place via messaging apps as well as phone calls, there are growing numbers of use-cases where chatbot\ndeployment gives organizations a clear return on investment. Call center workers may be particularly at risk from AI-driven\nchatbots.!85]\n\nChatbot jobs\n\nChatbot developers create, debug, and maintain applications that automate customer services or other communication processes.\nTheir duties include reviewing and simplifying code when needed. They may also help companies implement bots in their operations.\n\nA study by Forrester (June 2017) predicted that 25% of all jobs would be impacted by AI technologies by 2019.!86]\n\nSee also\n\n= Applications of artificial intelligence =» Autonomous agent\n\n8 of 18 2023-11-10, 9:32 a.m.", metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 8})

Document Transformation

We could work with the documents we have just as they are now, but it is sometimes difficult to get the proper level of context from a question from an entire page of information. One way to rectify this is by using the ReqcursiveCharacterTextSplitter function which will break down the data into smaller sections. The function has some built-in breakpoints point like section titles and ends of paragraphs. We could then use the smaller split up documents to makes sure our search’s are more accurate. I have found that using both the overall page documents for overall context and the smaller in-depth documents for smaller details together works best. We see that this drastically increases the number of documents that we need to search through.

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size = 512, chunk_overlap = 100)

documents = text_splitter.split_documents(docs)
documents = documents + docs
len(documents)

454

Text Embeddings

Text embeddings are the real power behind all natural language processes, including chatbots. A text embedding is a vector that represents the meaning behind a text and not the text itself. In this way, we can search through the meaning of our documents for an answer without needing to match the text exactly. In order to use the Open AI API to answer our questions, we need to utilize the Open AI Embeddings model. The use of it is pretty simple, when send them a document and they return the embedding.

import openai
from langchain.embeddings import OpenAIEmbeddings

openai.api_key = os.getenv("OPENAI_API_KEY")
embeddings_model = OpenAIEmbeddings(request_timeout = 60)

Vector Store

The next step is not 100% necessary, but it will make our lives much easier. What we need to do is to create a database that connects each of the documents with their embedding. We can do this by creating a vector store. There are a few different vector stores, the one I chose to us is `FAISS’. All we need to do is define the vector store, send it our stored documents and send the connection to our embedding model. The function will then process each document for us.

from langchain.vectorstores import FAISS

db = FAISS.from_documents(documents, embeddings_model)

MultiQuery Retriever

What we have now is a database that we can search for ourselves if we like. You can query the vector store, convert that query into an embedding, and search the vector store for matches. The return from this query will then be a list of documents in the vector store, which maybe useful, but can be difficult to interpret by itself. What a custom chatbot does is use this basic search results and supply it to a Large Language Model (llm) to make sense of it. We would like our Chatbot to have the entire vector store to look through to answer our questions, but there are some limitations to how much context we can provide to Open Ai.

For reference, the Chat GPT3 API has a limit of about 4000 tokens. What are tokens? Well, tokens are small sections of text. They are usually longer than 1 character but smaller than most words. One thing to keep in mind is that the 4000 token limitation covers everything, including context provided, the query and the answer returned. Do to this limitation, we need to do a search through our vector store and limit the context that we send to the Open AI API.

We could perform a basic query, but I prefer to create a Multi-query. In this query, we are actually asking ChatGPT to create different variations to our query. We can then search our vector store on multiple queries are return the best results over all queries. There is some expense to this as it requires an additional API call to generate the new queries.

We can then send our question to our retriever which will automatically perform our search over the vector store, I use the default prompt as it seems to do a good job. The custom Chat bot will then return the answer to our question, based on the context provided by the vector store search, with the source documents it used to generate an answer.

from langchain.chains import RetrievalQA
from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain.llms import OpenAI

retriever_from_llm = MultiQueryRetriever.from_llm(
  retriever = db.as_retriever(search = 'mmr', search_kwags = {'k': 10}),
  llm = OpenAI())
  
question = "What do Chat bots do?"
qa = RetrievalQA.from_chain_type(llm = OpenAI(),
chain_type = "stuff",
retriever = retriever_from_llm,
return_source_documents = True)

ans = qa({"query":question}, return_only_outputs = True)
print(ans)

{'result': ' Chatbots are software applications or web interfaces that aim to mimic human conversation through text or voice interactions. Modern chatbots are typically online and use artificial intelligence (AI) systems that are capable of maintaining a conversation with a user in natural language and simulating the way a human would behave as a conversational partner. They are used for B2C customer service, sales and marketing, as well as in toys, devices, and other applications. They can also be used to fill chat rooms with spam and advertisements, by mimicking human behavior and conversations or to entice people into revealing personal information.', 'source_documents': [Document(page_content='A chatbot (originally chatterbot!!) is a software application or web interface that aims to\nmimic human conversation through text or voice interactions.l2I[3Il41 Modern chatbots are\ntypically online and use artificial intelligence (AI) systems that are capable of maintaining a\nconversation with a user in natural language and simulating the way a human would behave as a\nconversational partner. Such technologies often utilize aspects of deep learning and natural', metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 1}), Document(page_content=': 3\n0: Chatbot - Wikipedia https://en.wikipedia.org/wiki/Chatbot\n\nIn 2016, Facebook Messenger allowed developers to place chatbots on their platform. There were 30,000 bots created for Messenger\nin the first six months, rising to 100,000 by September 2017.!27]\n\nSince September 2017, this has also been as part of a pilot program on WhatsApp. Airlines KLM and Aeroméxico both announced\ntheir participation in the testing;!281[29l[30I[31] oth airlines had previously launched customer services on the Facebook Messenger\nplatform.\n\nThe bots usually appear as one of the user\'s contacts, but can sometimes act as participants in a group chat.\n\nMany banks, insurers, media companies, e-commerce companies, airlines, hotel chains, retailers, health care providers, government\nentities and restaurant chains have used chatbots to answer simple questions, increase customer engagement,!32! for promotion, and\nto offer additional ways to order from them.!33]\n\nA 2017 study showed 4% of companies used chatbots.!34] According to a 2016 study, 80% of businesses said they intended to have\none by 2020,[35]\n\nAs part of company apps and websites\n\nPrevious generations of chatbots were present on company websites, e.g. Ask Jenn from Alaska Airlines which debuted in 200834 or\nExpedia\'s virtual customer service agent which launched in 2011.[361[37] The newer generation of chatbots includes IBM Watson-\npowered "Rocky", introduced in February 2017 by the New York City-based e-commerce company Rare Carat to provide information\nto prospective diamond buyers.!381[39]\n\nChatbot sequences\n\nUsed by marketers to script sequences of messages, very similar to an autoresponder sequence. Such sequences can be triggered by\nuser opt-in or the use of keywords within user interactions. After a trigger occurs a sequence of messages is delivered until the next\nanticipated user response. Each user response is used in the decision tree to help the chatbot navigate the response sequences to\ndeliver the correct response message.\n\nCompany internal platforms\n\nOther companies explore ways they can use chatbots internally, for example for Customer Support, Human Resources, or even in\nInternet-of-Things (IoT) projects. Overstock.com, for one, has reportedly launched a chatbot named Mila to automate certain simple\nyet time-consuming processes when requesting sick leave.[4°] Other large companies such as Lloyds Banking Group, Royal Bank of\n\n4 of 18 2023-11-10, 9:32 a.m.', metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 4}), Document(page_content='of chatbots). Chatbots can also be designed or customized to further target even more specific\nsituations and/or particular subject-matter domains.!Z]', metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 1}), Document(page_content=': 2\n0: Chatbot - Wikipedia https://en.wikipedia.org/wiki/Chatbot\n\n(though the program as released would not have been capable of doing so).{15]\n\nFrom 197816] to some time after 1983,!7] the CYRUS project led by Janet Kolodner constructed a chatbot simulating Cyrus Vance\n(57th United States Secretary of State). It used case-based reasoning, and updated its database daily by parsing wire news from\nUnited Press International. The program was unable to process the news items subsequent to the surprise resignation of Cyrus Vance\nin April 1980, and the team constructed another chatbot simulating his successor, Edmund Muskie.8]57]\n\nOne pertinent field of AI research is natural-language processing. Usually, weak AI fields employ specialized software or\nprogramming languages created specifically for the narrow function required. For example, A.L.I.C.E. uses a markup language called\nAIML,3] which is specific to its function as a conversational agent, and has since been adopted by various other developers of, so-\ncalled, Alicebots. Nevertheless, A.L.I.C.E. is still purely based on pattern matching techniques without any reasoning capabilities, the\nsame technique ELIZA was using back in 1966. This is not strong AI, which would require sapience and logical reasoning abilities.\n\nJabberwacky learns new responses and context based on real-time user interactions, rather than being driven from a static database.\nSome more recent chatbots also combine real-time learning with evolutionary algorithms that optimize their ability to communicate\nbased on each conversation held. Still, there is currently no general purpose conversational artificial intelligence, and some software\ndevelopers focus on the practical aspect, information retrieval.\n\nChatbot competitions focus on the Turing test or more specific goals. Two such annual contests are the Loebner Prize and The\nChatterbox Challenge (the latter has been offline since 2015, however, materials can still be found from web archives)./291\n\nChatbots may use neural networks as a language model. For example, generative pre-trained transformers (GPT), which use the\ntransformer architecture, have become common to build sophisticated chatbots. The "pre-training" in its name refers to the initial\ntraining process on a large text corpus, which provides a solid foundation for the model to perform well on downstream tasks with\nlimited amounts of task-specific data. An example of a GPT chatbot is ChatGPT.[2°] Despite criticism of its accuracy, ChatGPT has\ngained attention for its detailed responses and historical knowledge. Another example is BioGPT, developed by Microsoft, which\nfocuses on answering biomedical questions.!21I[22]\n\nDBpedia created a chatbot during the GSoC of 2017./231[241I25] Tt can communicate through Facebook Messenger.\n\nApplication\nMessaging apps\nMany companies’ chatbots run on messaging apps or simply via SMS. They are used for B2C customer service, sales and\n\nmarketing. [26]\n\n3 of 18 2023-11-10, 9:32 a.m.', metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 3}), Document(page_content=': 2\n0: Chatbot - Wikipedia https://en.wikipedia.org/wiki/Chatbot\n\n(though the program as released would not have been capable of doing so).{15]', metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 3}), Document(page_content=": 6\n0: Chatbot - Wikipedia https://en.wikipedia.org/wiki/Chatbot\n\nIn India, the state government has launched a chatbot for its Aaple Sarkar platform,!Z] which provides conversational access to\ninformation regarding public services managed.!721I73]\n\nToys\nChatbots have also been incorporated into devices not primarily meant for computing, such as toys.[74]\n\nHello Barbie is an Internet-connected version of the doll that uses a chatbot provided by the company ToyTalk,!75] which previously\nused the chatbot for a range of smartphone-based characters for children.!76] These characters' behaviors are constrained by a set of\nrules that in effect emulate a particular character and produce a storyline.!77]\n\nThe My Friend Cayla doll was marketed as a line of 18-inch (46 cm) dolls which uses speech recognition technology in conjunction\nwith an Android or iOS mobile app to recognize the child's speech and have a conversation. It, like the Hello Barbie doll, attracted\ncontroversy due to vulnerabilities with the doll's Bluetooth stack and its use of data collected from the child's speech.\n\nIBM's Watson computer has been used as the basis for chatbot-based educational toys for companies such as CogniToys!74] intended\nto interact with children for educational purposes.!78]\n\nMalicious use\n\nMalicious chatbots are frequently used to fill chat rooms with spam and advertisements, by mimicking human behavior and\nconversations or to entice people into revealing personal information, such as bank account numbers. They were commonly found on\nYahoo! Messenger, Windows Live Messenger, AOL Instant Messenger and other instant messaging protocols. There has also been a\npublished report of a chatbot used in a fake personal ad on a dating service's website.[79]\n\nTay, an AI chatbot that learns from previous interaction, caused major controversy due to it being targeted by internet trolls on\nTwitter. The bot was exploited, and after 16 hours began to send extremely offensive Tweets to users. This suggests that although the\nbot learned effectively from experience, adequate protection was not put in place to prevent misuse.[8°]\n\nIf a text-sending algorithm can pass itself off as a human instead of a chatbot, its message would be more credible. Therefore,\nhuman-seeming chatbots with well-crafted online identities could start scattering fake news that seems plausible, for instance\nmaking false claims during an election. With enough chatbots, it might be even possible to achieve artificial social proof.[81J[82]\n\nLimitations of chatbots\n\n7 of 18 2023-11-10, 9:32 a.m.", metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 7}), Document(page_content='Chatbots and jobs\n\nChatbots are increasingly present in businesses and often are used to automate tasks that do not require skill-based talents. With\ncustomer service taking place via messaging apps as well as phone calls, there are growing numbers of use-cases where chatbot\ndeployment gives organizations a clear return on investment. Call center workers may be particularly at risk from AI-driven\nchatbots.!85]\n\nChatbot jobs', metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 8}), Document(page_content='Many high-tech banking organizations are looking to integrate automated AI-based solutions such as chatbots into their customer\nservice in order to provide faster and cheaper assistance to their clients who are becoming increasingly comfortable with technology.\nIn particular, chatbots can efficiently conduct a dialogue, usually replacing other communication tools such as email, phone, or SMS.', metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 5}), Document(page_content='Chatbot jobs\n\nChatbot developers create, debug, and maintain applications that automate customer services or other communication processes.\nTheir duties include reviewing and simplifying code when needed. They may also help companies implement bots in their operations.\n\nA study by Forrester (June 2017) predicted that 25% of all jobs would be impacted by AI technologies by 2019.!86]\n\nSee also\n\n= Applications of artificial intelligence =» Autonomous agent\n\n8 of 18 2023-11-10, 9:32 a.m.', metadata={'source': 'C:\\Users\\Mark\\Desktop\\csv_files\\Chatbot - Wikipedia.csv', 'row': 8})]}

If you are wondering what the default query looks like, you can use the following code. You can create your own prompt if you like, but the default seems to work well for me. With a custom prompt you can customize the output, like make the results return summarized in a Markdown table if you like.

print(retriever_from_llm.llm_chain.prompt)

input_variables=['question'] template='You are an AI language model assistant. Your task is \n    to generate 3 different versions of the given user \n    question to retrieve relevant documents from a vector  database. \n    By generating multiple perspectives on the user question, \n    your goal is to help the user overcome some of the limitations \n    of distance-based similarity search. Provide these alternative \n    questions separated by newlines. Original question: {question}'

Conclusion

Well, there are all the steps needed to create your very own custom ChatGPT bot with Lang chain. We loaded our CSV files, split the documents into smaller section, created a vector store of our documents, search through our vector store with a multi-query and forward the context and our question to the Open AI API. The result is an answer that is limited to the data stored in our PDF documents.

Photo by Google DeepMind

Introduction

I have recently completed a project at work, the creation of a custom ChatGPT chatbot. I will break the project into two parts, the first part will scan a folder of PDF files into a dataframe and the second part will pass the data to OpenAI API. This entire project was completed in python.

Project outline

PDFs can be easily scanned in python with the pypdf module. It is easily installed and easily run, but I have found that the quality of the scan to be lacking. Pypdf also seems to have some issues with PDFs that created from scanned documents, not directly created from a text document. For this reason, I have found an alternative method.

The first step is to convert all the PDFs in a directory to PNG images. This can be achieved with the convert from path function from the pdf2image library with the poppler application. The poppler program can be downloaded here and unzipped into its own directory. There is no Windows version on the poppler site, but I found a repo with a nearly updated Windows version here. You will need to copy the directory path into your code. We can create a for loop to open each PDF file one at a time. It’s important to remember to change the ‘' to’/’ for Windows users when referring to directory positions.

The second step is to then scan through the PNG images with OCR. For this task, we can use Tesseract. Tesseract is a Google project that is easy to use. Like Poppler, you will need to download the application separately. You will also need to install the helper python package pytessseract. The Tesseract application can be found here. I have my program to save the data in a CSV file, but you can store it anyway you want. I decided to save each PDF file as a separate CSV file and assigning each row as a different PNG file or page of the PDF. This was to ensure that may data is easily organized

The next stages require getting into the Langchain library. These steps will be included in the follow-up to this post as both post, are quite lengthy and each can stand alone.

Converting PDF to PNG with Poppler

Again, prior to running this code, you will need to install the Poppler Application. You also need to copy the directory to the location of the Poppler bin folder. The rest of this section is pretty simple, I’ve created a loop to go through every filename that ends with ‘.pdf’ in a specific PDF folder. I also save the PNG file with the page number included into the title. If the results from the OCR scans are inaccurate, you can adjust the resolution of the PNG files with the parameter ‘dpi = 300’ passed to the convert from path function. The default value is 100. Fair warning, increasing the resolution will slow down the entire process and can potentially add additional artifacts into the OCR scan.

import os
import pandas as pd
from PIL import Image
from pdf2image import convert_from_path

poppler_path = 'C:/Program Files/poppler-23.08.0/Library/bin'
for pdf_file in [f for f in os.listdir('//Desktop/PDF') if f.endswith('.pdf')]:
  images = convert_from_path(pdf_path = '//Desktop/PDF/' + pdf_file, poppler_path = poppler_path)

  for count, img in enumerate(images):
    img_name = f"{pdf_file[:-4]}_page_{count}.png"
    img.save('//Desktop/PDF/' + img_name, "PNG")

OCR from PNG files

The Tesseract application is required for the next stage. Since every PNG from every PDF will need to go through the process, I’ve recreated the first section and included the Tesseract functions into the same loop. I’ve also included a step to delete each PNG file after it has been scanned, since it will no longer be needed. The final stage is to save all the returned data as a CSV file. I have found that it is useful to specify the encoding used in saving the CSV.

import os
import pandas as pd
from PIL import Image
from pdf2image import convert_from_path
import pytesseract

poppler_path = 'C:/Program Files/poppler-23.08.0/Library/bin'
pytesseract.pytesseract.tesseract_cmd = '//Tesseract-OCR/tesseract.exe'

for pdf_file in [f for f in os.listdir('//Desktop/PDF') if f.endswith('.pdf')]:
  images = convert_from_path(pdf_path = '//Desktop/PDF' + pdf_file, poppler_path = poppler_path)
  extracted_text = []
  for count, img in enumerate(images):
    img_name = f"{pdf_file[:-4]}_page_{count}.png"
    img.save('//Desktop/PDF' + img_name, "PNG")
    
    extracted_data.append(pytesseract.image_to_string(Image.open('C:/Users/Mark/Desktop/PDF' + img_name)))
    os.remove('//Desktop/PDF' + img_name)
    
  df = pd.DataFrame(extracted_text)
  df.to_csv('//Desktop/PDF' + pdf_name[:-4] + '.csv', encoding = 'utf-8-sig')

Conclusion

We are finally able to create a usable CSV file from a OCR scanned PDF file. The first step was to convert the pdf into PNG files with Poppler. Each png is then scanned with Tesseract. And the returned values are stored in a CSV file. By why would you want to go through all the steps in the first place? Well, we will need to proceed with the next post about creating the ChatGPT chatbot.

Introduction

I was recently reading my copy of “An Introduction to Statistical Learning” (my Amazon affiliate link) and got the chapter about the different tree based methods. I am pretty familiar with Random Forest, but a few of the other methods are new to me. Let’s explore these different techniques.

For these examples, I will explore the glass dataset from the openml site. This dataset has 9 different features used to predict the type of glass. The dataset has 214 observations.

The dataset is downloaded with the following code. This requires the farff package to open the arff files used on the openml site.

library(tidyverse)
download.file("https://www.openml.org/data/download/41/dataset_41_glass.arff", "data.arff")
df <- farff::readARFF("data.arff")

From there, we can start to explore the dataset and set up a train and testing split for the data.

df %>% str()

'data.frame':   214 obs. of  10 variables:
 $ RI  : num  1.52 1.52 1.52 1.51 1.53 ...
 $ Na  : num  12.8 12.2 13.2 14.4 12.3 ...
 $ Mg  : num  3.5 3.52 3.48 1.74 0 2.85 3.65 2.84 0 3.9 ...
 $ Al  : num  1.12 1.35 1.41 1.54 1 1.44 0.65 1.28 2.68 1.3 ...
 $ Si  : num  73 72.9 72.6 74.5 70.2 ...
 $ K   : num  0.64 0.57 0.59 0 0.12 0.57 0.06 0.55 0.08 0.55 ...
 $ Ca  : num  8.77 8.53 8.43 7.59 16.19 ...
 $ Ba  : num  0 0 0 0 0 0.11 0 0 0.61 0 ...
 $ Fe  : num  0 0 0 0 0.24 0.22 0 0 0.05 0.28 ...
 $ Type: Factor w/ 7 levels "build wind float",..: 1 3 1 6 2 2 3 1 7 2 ...

df <- droplevels(df)

set.seed(1234)
test <- sample(1:nrow(df), floor(nrow(df)*0.2))

testdf <- df[test,]
traindf <- df[-test,]

Decision Tree

The most basic model is the decision tree. In this model, all classes are stored in a container at the root of the tree. The root is split by certain feature values into two smaller containers. These smaller containers are called leafs. After these initial two leafs, we perform an additional split resulting in 4 new leafs. At this point, there should be some better sorting on the response value in the new leafs. Splitting and creating new leafs is continued until the desire results are obtained.

The main disadvantage of decision trees is that they are very prone to over fitting. It is pretty easy to imagine a tree that has split so many times that each leaf now represents a single observation. This tree would have 100% accuracy on the training set, but would not perform well on a test set. We prevent over fitting by cutting off or pruning the number of leafs to a certain value.

For our example, the decision tree is created with the tree package. It is pretty simple to use, you just supply it with the function and the data.

mdltree <- tree::tree(Type ~., data = traindf)

A tree visualization is easily created with the print command. With an extra line, we can add the text that shows where each leaf is split. It does get difficult to understand the splits when getting to the lower level leafs.

mdltree %>%
plot(mdltree)
text(mdltree, cex =.4)

We can now test the decision tree’s performance on the test data. Since the response is a factor, the resulting prediction is a matrix with a probability assigned to each response for every observation. We can summarize the mostly likely response with the max.col function.

treevalues <- predict(mdltree, newdata = testdf)
treepred <- colnames(treevalues)[max.col(treevalues)]
Acctree <- mean(treepred == testdf$Type)
Acctree

[1] 0.5

From this prediction, we can see that the decision tree didn’t provide a high level of accuracy (50% Accuracy). For a higher level of accuracy, let’s explore some additional methods.

Bagging

Bootstrapping is a useful technique when we have a limited number of observations. Ordinarily, we take each training observation out of a bag and train our model with it. In bootstrapping, after we train our model with an observation, we put the observation back in the bag, allowing repeated selection. The repeated selection creates a smoother distribution of observations.

Bagging takes this process one step further. We take multiple models from bootstrapped training data and average their values. Each bootstrapped model is unique because the training observations are randomly selected.

Bagging can be performed with the randomForest function. By default, the function creates trees from a subset of features, but we can create bagged trees by using all features. The default behaviour for a random forest is to take the majority vote for constructed trees or average their prediction.

library(randomForest)
mdlbag <- randomForest(Type ~., data = droplevels(traindf), mtry = ncol(df)-1, n.trees = 500)
bagpreds <- predict(mdlbag, newdata = testdf)
Accbag <- mean(bagpreds == testdf$Type)
Accbag

[1] 0.6666667

Random Forest

A random forest is an ensemble model, meaning the prediction is created by combining multiple models together. It is very similar to the previous example with Bagging. The main difference is that the Random Forest selects a random set of features when creating each tree.

The rationale for a random feature selection is to create more unique trees. If there is a strong feature in bagging, then most of the trees will use it to create their first split. This creates trees that appear very similar.

mdlforest <- randomForest(Type ~., data = droplevels(traindf), n.trees = 500)
rfpreds <- predict(mdlforest, newdata = testdf)
Accrf <- mean(rfpreds == testdf$Type)
Accrf

[1] 0.6666667

Boosting

Boosting is another multiple use technique, much like bagging. Instead of effecting the train data to create different trees, boosting builds trees in sequential order with the default training set. Rather than fitting trees to the response, trees are fit to their residuals. Each tree then learns slowly with the residuals from the previous tree.

For this model, we will to use two libraries, caret and gbm. The gbm package is the package that provides the actual model, but a warning is displayed when using the gbm function. To get around the issue, we can use the caret package with the train function. This function can accept many different models and will automatically go through tuning parameters.

library(caret)
library(gbm)
mdlboost <- caret::train(Type ~., data = traindf, method = 'gbm', verbose = FALSE)
boostpreds <- predict(mdlboost, newdata = testdf)
Accboost <- mean(boostpreds == testdf$Type)
Accboost

[1] 0.6904762

Bayesian Additive Regression

Bayesian Additive Regression Trees (BART) are similar to the previously mentioned models, creating trees with some random elements that model the signal not captured in previous trees.

In the first iteration of the model, all trees are initialized to the same root node. After other iteration updates each of the kth trees one at a time. During the bth iteration, we subtract the predicted values from each response to produce a partial residual. i represents each iteration.

Rather than fitting a new tree to this residual, BART randomly chooses a tree from the previous iteration (), favouring trees that improve fit.

Unfortunately, I have not been able to get any BART model to work for this multi-class classification example. The bartMachine package doesn’t have an implementation, and the BART package’s implementation doesn’t seem to work for me. All the values for me were returning NA. There is also very little resources on the BART package to troubleshoot.

Conclusion

The decision tree as an easy-to-understand model, but they are prone to over fitting data. Pruning is required to reduce this, but will also reduce accuracy(50% Accuracy for this example).

Ensemble models are then introduced with Bagging which averages a bunch of trees created by a selection of different observations. This does increase our accuracy(66.67%). This is very similar to the Random Forest model, but instead we increase the random element to include the features selected (66.67%).

The final models create trees that build of previously generated trees with the boosting and the Bayesian Additive Regression Tree. Unfortunately, there are no results for the BART model, but boosting does produce good results (69.05%).

So there is no clear answer which model is the best, they follow pretty similar methods and produce similar results. I would recommend training them all and finding which performs best on your data.

Introduction

I have officially switched my blog from blogdown to quarto due to the new features that quarto adds. The move has not been painless, but most of the most important features remain the same. The feature this is currently missing in quarto that I cannot live without, is the 1 click creation of a new blog post. There are a few different ways to automate the creation of blog posts, here are a few that I’ve found.

RMD Template

It is fairly simple to create a new RMD template for the markdown package. These templates are available when you click File -> New File -> R Markdown… under the tab ‘from template’. There are many templates available under the rmarkdown package, but you might have more templates from other packages. Unfortunately, this feature is not yet available in quarto for QMD files.

So, currently there is no way to create a QMD template, but there isn’t a major difference between QMD and RMD files. An easy solution would be to create a RMD template with built-in quarto commands and when you use it to create a blog post you just save it as a QMD file. This will still require going through the steps of creating a RMD template.

Creating a RMD template is pretty simple, it just requires creating two files and a folder. This is even easier if you use the usethis package with its use_rmarkdown_template function, as it will automatically create the structure.

Within the main template folder, you will have a ‘skeleton’ folder and a ‘template.yaml’ file. The YAML file will contain the name and description of the template. It also includes the parameter create_dri which will create a file in a nested folder if the option is true. The structure of the file is the following:

name: Quarto Blog Post
description: >
  Template for Quatro Blog post. 
create_dir: true

Within the ‘skeleton’ folder, you will find a ‘skeleton.RMD’ files. This is the actual template structure. This file should have all the YAML, including QMD features, and the basic structure of your blog post. The following is a sample that I have created:

---
title: 'Post for Quarto Blog'
author: Mark Edney
date: ""
categories:
  - How-to
  - R
  - Rmarkdown
  - Python
draft: true
description: ''
image: ""
archives:
  - ""
toc: false

format: 
  html: 
    code-fold: false
    code-tools: false
    code-link: false
---

With the template setup, all you need to do is add it to a package. You can add it to any package, but I would recommend adding it to the rmarkdown package with the rest of its templates. The directory for this package in windows can be found somewhere like this: ‘C:~login-library~R version’ With the folder moved over, you should be able to see your template listed when you create a RMD file after you restart R Studio.

This is a pretty easy way to create a quarto blog post, but there are additional steps required. You will need to manually type the date, saved the file as a QMD, and create whatever folder structure you like for your quarto blog. So, let’s explore more options to make it even easier.

Whisker package

There are a few different packages available to create template files like the brew and whisker packages. I decided to try the whisker package as it seemed to be the easiest to learn. In the whisker, you create a template and refer to your parameters in the ‘{{}}’ set of brackets.

I decided to create a function that will accept the user input for the blog and apply it to the template. The function will then create the proper directory layout that I use for my blog posts.

data <- list(title = title,
               author = 'Mark Edney',
               date = Sys.Date(),
               categories = list("How-to", "R", "Rmarkdown"),
               draft = 'true',
               description = "''",
               image = "''",
               archives = format(date, "%Y/%m"),
               toc = 'false',
               fold = "show",
               tools = 'true',
               link =  'false')

Template <- '---
title: {{title}}
author: {{author}}
date: {{date}}
categories: [{{categories}}]
draft: {{draft}}
description: {{description}}
image: {{image}}
archives:
  - {{archives}}
toc: {{toc}}

format:
  html: 
    code-fold: {{fold}}
    code-tools: {{tools}}
---

# Introduction

# Conclusion
'

Blog_post <- function(title){
  data <- list(...
  
  Template <- '---
title: {{title}}...

  dir.create(paste0("./posts/",data$date, "-", title))
  writeLines(whisker.render(Template, data),
  paste0("./posts/",data$date, "-", title, "/index.qmd"))
}

Addins

Addins are a special feature of RStudio with its only icon on the toolbar. This icon allows the user to run a set of R code from the toolbar. In order to create an Addin, you first need to create a package that includes all the code.

Package: Qpostr
Type: Package
Title: Create a Quarto Blog Post
Version: 0.1.0
Author: Mark Edney
Maintainer: Mark Edney <m2edney@gmail.com>
Description: This package is used to create an addin for creating a new Quarto blog post. This post is created using a default template using the whisker pakage. A Shiny gadget is utilized to get user input prior to the creataion of the blog post. 
License: Creative Commons Attribution-NonCommercial-NoDerivs 3.0 United States License
Encoding: UTF-8
LazyData: true

\name{QGadget}
\alias{}
\title{Create a Qaurto Blog Post}
\usage{
Qgadget()
}
\description{
Creates a Quarto Blog Post in the posts directory from a template. Opens a Shiny Gadget to get user input before creating the post. Gadget is required for creating the addin.
}

Name: Quarto Blog Post
Description: Creates a new Quarto Blog post from template
Binding: QGadget
Interactive: true

QGadget <- function() {....}

ui <- miniUI::miniPage(
  miniUI::gadgetTitleBar("Quarto Blog Post"),
  miniUI::miniContentPanel(
    shiny::textInput("title", "Title", placeholder = "Post Title"),
    shiny::textInput("author", "Author", placeholder = "")
    )
  )

server <- function(input, output, session) {
  
  shiny::observeEvent(input$done, {
    Blog_post(input$title, input$author)
      stopApp("Post Created")
      })
  }

shiny::runGadget(ui, server, viewer = shiny::dialogViewer("Quarto Blog Post"))

Blog_post <- function(title, author){
  data <- list(title = title,
               author = author,
               date = Sys.Date(),
               categories = list("How-to", "R", "Rmarkdown"),
               draft = 'true',
               description = "''",
               image = "''",
               archives = format(date, "%Y/%m"),
               toc = 'false',
               fold = "show",
               tools = 'true',
               link =  'false')

  Template <- '---
title: {{title}}
author: {{author}}
date: {{date}}
categories: [{{categories}}]
draft: {{draft}}
description: {{description}}
image: {{image}}
archives:
  - {{archives}}
toc: {{toc}}

format:
  html:
    code-fold: {{fold}}
    code-tools: {{tools}}
---

# Introduction

# Conclusion

'

  dir.create(paste0("./posts/",data$date, "-", title))
  writeLines(whisker::whisker.render(Template, data), paste0("./posts/",data$date, "-", title, "/index.qmd"))
  file.edit(paste0("./posts/",data$date, "-", title, "/index.qmd"))
}

Conclusion

In summary, there is no native way to create a new quarto post at this time. We have explored multiple different solution

• Create a RMD template make a document and rename it as a QMD
• Use the whisker package to create template which accepts calculated fields
• Store the whisker template in a function stored in a script file, or to be loaded by default in the .Rprofile file
• Create an non-interactive addin by storing the function in a package
• Create an interactive addin by making a Shiny Gadget.

Personally, my go to answer would be to use the interactive addin. If you are interested in installing the package I have create, feel free to use the following code to install it. After it is installed, you will need to restart R for the addin to be in the addin tab.

devtools::install_github('mark-edney/Qpostr')

Setup

The setup for Quarto is pretty simple. You will need to visit the quarto website to download the Quarto Command Line Interface (CLI). There are step-by-step instructions for your selected text editor. I am most familiar with RStudio for R and VSCode for Python.

For Rstudio, it’s pretty much just plug and play now. I did install the Quarto package, but all the commands can be done by the command line interface. Switching from RMarkdown is as simple as saving them as qmd file. The process for Quarto for RStudio can be described by the following process flow:

It is not much more difficult for VSCode, all you need to do is download the Quarto extension. The process flow is similar to RStudio but uses Jupyter instead of knitr.

With the setup complete, there should be no differences between text editors.

Code Chunk Options

The first new feature to explore the support for code chuck options within the code chunks. These options would usually live within the code chunk title line. Any supported option can be added with the #| tag. This feature is useful for situations with many options, as it does increase readability.

```{r}
#| label: load
#| include: true
#| warning: false

library(tidyverse)
data("msleep")
```

Code-folding

One of the neat new features is code-folding. When this feature is enabled in the qmd YAML, the person viewing the document can hide/unhide code chunks. This can make it easier for them to read the document. Only the code will be hidden, and not the results.

glimpse(msleep)

This feature is enabled by making the following addition to the YAML. You would change the format from HTML to your required format, such as PDF.

format: 
  html: 
    code-fold: true
    code-tools: true

With the addition of the code-tools: true parameter, the reader can decide to hide all code chunks from the top of the document.

Figures

Quarto provides a bunch of additional tools for displaying figures. You can assign values for captions, sub-captions, width and height. You can even create a figure with multiple plots with separate sub-captions.

```{r}
#| label: fig-sleep
#| fig-cap: "Sleeping habits of animals"
#| fig-subcap:
#|   - "Scatter plot of body weight by total sleep"
#|   - "Violin plot of REM sleep by vore"
#| layout-ncol: 2

msleep %>%
  drop_na(sleep_total, bodywt) %>%
  ggplot(aes(y= sleep_total, x = bodywt)) +
  geom_point(color = "blue") +
  theme_minimal()

msleep %>%
  group_by(vore) %>%
  drop_na(sleep_rem, vore) %>%
  ggplot(aes(y= sleep_rem, x = vore)) +
  geom_violin(aes(fill = vore)) +
  theme_minimal()
```

You can now use cross-referencing for the figure by referencing the figure. This means that in your text, you can refer to the figure number and link to the figure. This will automatically update your figure numbers and is achieved by typing the ‘@’ symbol followed by the figure label. As an example, ‘@fig-sleep’ turns into Figure 1.

There is an additional option to let the figures take up the width of the entire page, but I would not recommend using it as it extends beyond the width of the body of your page. It requires the following code:

```{r}
#| column: page
```

Code Linking

A reader may not be familiar with all the functions that you use in your document, so it may be useful to enable code linking. With code linking, a function in a code chunk will have a hyperlink to the documentation for that function. To work in R, this feature requires the xml2 and downlit packages.

lm(as.factor(order) ~ sleep_total, data = msleep[complete.cases(msleep),] )

Table of contents

I think the best feature for Quarto is the floating table of contents. I can’t describe how much time and effort I’ve spent trying to get a floating table of contents in a Blogdown blog. It didn’t work for me, it would require getting deep into the weeds changing the CSS layout for my HUGO theme. It was not worth the effort.

Adding a floating table of contents in Quarto is simple. Just use the following code in the document YAML:

toc: TRUE

One simple line of code in the YAML and your document has a floating table of contents. There is some additional customization such as the level of headers, location and title.

toc: true
toc-depth: 2
toc-location: left
toc-title: Contents

Quarto vs Blogdown

With my experimentation with Quarto, I decided to move my blogdown blog to Quarto. In theory, this should be a simple switch, with just copying all post from folder to another. Quarto can use rmd files, but they can easily be changed over to qmd files. I decided to switch all my post to the qmd format and include some additional features. The Quarto site has extensive reference information for creating a blog. (“Quarto,” n.d.-)

I did have an issue with one of my post not rendering correctly. This maybe an issue with compatibility with the stargazer package. In the end, I decided to just remove the post altogether as I could get it to render correctly, and I prefer the gt over the stargazer package for creating good-looking tables.

Conclusion

It is easy to create great looking documents using quarto, whether that be with code in python or R. Quarto supports most of the features in RMarkdown with some fancy new ones. My personal favourite is the floating table of contents. I have also found that rendering a Quarto blog is a much smoother experience than rendering a blogdown blog.

Introduction

Network graphs are an important tool for network analysis. They illustrate points, referred to as nodes, with connecting lines, referred to as edges. Since network graphs are such useful tools, there are many options for graph generation. In this posting, I will demonstrate three different techniques for developing network graphs in r.

This is part 3 of a series which is based on the Stormlight Archive by Brandon Sanderson. This project was originally inspired by the work of Thu Vu where she created a network mapping of the characters in the Witcher series.

In the first part of the project, we scrapped the Coopermind website to create a verified character name list. This scrapping was performed with the rvest package. The list was then cleaned up and saved for further use.

For the second part of the project, we read through and analyzed the four books that make up the Stormlight Archive series. The books were read into memory with the readtext package, which fed nicely into the quanteda to create the body of text called a Corpus. Unfortunately, the body of text was so big that we were unable to model all the text, so we divided the Corpus up into smaller documents with the rainette package.

With the corpus finally prepped, we feed it into the spacyr package, a frontend for the spaCy python library, to identify the entities. We were able to create a table identifying the entities that were people and filter it by the verified character list. We created a moving window model that would create a connection between two named characters if they were both mentioned within the same window. By aggregating the results of this model, we developed the foundation for a network graph.

Initialization

The first step of this process is to load in the necessary packages for the graph generation. The Tidyverse package is always useful for analysis, so I’ve loaded it too. I have read that the different graph packages can interrupt each other, requiring one of them to be loaded at a time. I have not found this to be an issue.

library(tidyverse)
library(igraph)
library(ggraph)
library(networkD3)

The next step is to load in the data that we created in part two of the project. This data represents that relationship between all the verified characters as read through the series of books. Saving and loading data in RDS format is much more convenient than the CSV format, as RDS files are compressed and seem to load faster.

data <- read_rds("StormGraph.RDS")

IGraph

The first package to explore is the igraph package. This package is not only for plotting graphs, but also includes many tools for network analysis. For our data, we can create a simple network graph with the graph_from_data_frame function. The relationships are not directional, so we pass this information to the function. The graph can then be plotted with the plot function.

graph <- graph_from_data_frame(data, directed = FALSE)
plot(graph)

The graph created is a mess. There are way too many character nodes and way too many relationships created. We need to create a smaller dataset to reduce the amount of information. I reduced the size of the data by taking only the top 98% quantile in relationships. Since the data is stored as a data table, the data table notation is used to create a subset.

data2 <- data[data$N >= quantile(data$N, p = 0.98),,]
data2 %>%
        graph_from_data_frame(directed = FALSE) %>%
        plot(layout = layout_with_graphopt)

The plot created is still difficult to understand, but it much more reasonable. I feel the igraph package is best for graph analysis and exploratory plots. For a more attractive plot, we need to move on to the next package.

Tidygraph and GGraph

The tidygraph and ggraph packages seek to create graphs in the tidyverse-like environment.

library(tidygraph)

Creating a graph with ggraph requires more structure than the previous igraph. The graph requires two data frames, one for nodes and one for edges.

For the nodes dataframe, we need a list of all the node names and an ID number for each node. This is achieved by finding the unique values within both columns of data. These values are then passed to the tibble function to create a tibble, a data structure similar to data frames, and then a column for IDs is created with the rowid_to_column function.

nodes <- c(data2$Person1, data2$Person2) %>% 
        unique() %>%
        tibble(label = .) %>%
        rowid_to_column("id")

For the edges dataframe, we need some additional steps. As a reminder, in our subset of data, we have rows with two names and a number to represent the strength of their bond. The character names need to in the form of the node IDs rather than the names. This task is completed with two merges with the node dataframe. The graph can then be created with the tbl_graph function.

edges <- data2 %>%
        left_join(nodes, by = c("Person1"="label")) %>%
        rename(from = "id") %>%
        left_join(nodes, by = c("Person2"="label")) %>%
        rename("to" = "id") %>%
        select(from, to, N)

graph_tidy <- tbl_graph(nodes = nodes, edges = edges, directed = FALSE)

For the plotting of the graph, we use the ggraph library. With this package, the graph can act as any other ggplot geom. With an extra step, we can create a centrality feature in our graph. There are a bunch of different centrality measures, but they all represent the level of importance of a node.

graph_tidy %>%
        mutate(Centrality = centrality_authority()) %>%
        ggraph(layout = "graphopt") + 
        geom_node_point(aes(size=Centrality, colour = label), show.legend = FALSE) +
        geom_edge_link(aes(width = N), alpha = 0.8, show.legend = FALSE) + 
        scale_edge_width(range = c(0.2, 2)) +
        geom_node_text(aes(label = label), repel = TRUE)

Network D3

The ggraph has created a better looking plot with a much higher level of customization. It is however a static plot with no level of interaction. I have tried using the ggplotly function from the plotly package it make it more interactive, but many of the ggraph features are not supported.

To create an interactive plot, we move to the networkD3 package. This package is based on the D3 JavaScript library to create interactive plots. We can use the same nodes and edges data frames from the ggraph plot. This process does require one adjustment to the node IDs, as the package requires an initial ID of 0 rather than the default r index of 1.

The function from the tidygraph, centrality_authority, is only supported for the tidygraph data structure, so we need an alternative function to use with our data frame. This is achieved with the authority.score function from the igraph package. Besides that, we normalize the edge width values, node sizes and set all the parameters for the forceNetwork function.

edges <- edges %>%
        mutate(from = from -1, to = to - 1) %>%
        mutate(N = N / 200)

nodes <- nodes %>%
        mutate(id=id-1) %>%
        mutate(nodesize = authority.score(graph_tidy)$vector*150)
        
forceNetwork(Links = edges, Nodes = nodes, Source = "from", Target = "to", NodeID = "label", Group = "id", opacity = 1, fontSize = 14, zoom = TRUE, Value = "N", Nodesize = "nodesize", opacityNoHover = TRUE)

Introduction

So in a previous post, we created a list of characters which will represent the nodes in our network graph. The next step in the project is to create the edges. The edges represent the relationships between characters. In our graph, we are going to have the edges represent that strength of the relationships between characters. In order to determine these edge values, we will need to perform relationship extraction from the text with the spacyr package.

The spacyr package is simply a wrapper for the python spaCy library, with the following functionality:

• tokenization
• lemmatizing tokens
• parsing dependencies (to determine grammatical structure)
• extracting form named entities

It uses the reticulate to create the python environment. I have previously written a post about using the reticulate package for using python code in RMarkdown.

Initialization

We start with the loading of the necessary libraries to complete the project.

library(spacyr)
library(tidyverse)
library(data.table)
#necessary to create a corpus object
library(readtext)
library(quanteda)
library(rainette)

If you have an environment of python with a version a spaCy, you can pass the destination into the spacy_intialize function. If not, you need to use the spacy_install function to create a Conda environment that will also include the spaCy package. For this project, I let spacyr create the Conda environment for me. This process did take a while for me, so don’t be surprised if it is the same for you.

spacy_install()

I have the name list from the web scraping post saved as a RDS files. RDS files are compressed text files which load quicker and take up much less space than a CSV file.

names <- read_rds("data/names.RDS")

Text Reading

The first step is to read all the text files into the system. I found this interesting little snippet of code that allows you to create a list of all the text files in a specific folder. For this project, all the books were stored in a single data folder.

list_of_files <- list.files(path = ".", recursive = TRUE,
                            pattern = "\\.txt$", 
                            full.names = TRUE)

With the list of files, we can use the map_df function from the purr package. The purr package is part of the tidyverse package, so we don’t need to load it separately. The map series of functions allows use to pass a vector of values and a function. Each value will then be passed to that function. The _df part of the function is just the requirement that the output be in the format of a dataframe.

The same task can be completed with a for loop, but it is much faster in the map function as it utilize vectorization. Vectorization is the strategy of performing multiple operations rather than a single operation at the same time. I am not very familiar with the purr package, so I plan to write a new article on the topic in the near future.

After all the books are read into memory, we need to create a corpus. A corpus is a large body of text, much like a library, for the sorting and organization of books. This is completed with the corpus function from the quanteda package. This corpus structure is necessary to utilize functions from the spacyr package.

The organizational structure in the Corpus is why I needed to load the books in with the readtext function from the readtext package. I’ve tried many methods to read the text(readlines, read_Lines, readfile) but none of them performed the proper way for the corpus function. There were plenty of issues, hours of difficulty, which resulted in me referring to the quanteda package website. There I learnt about the readtext function and it worked flawlessly on the first time. Well, I did find an issue with the default encoding not interpreting characters correctly, but this was corrected easily.

When the time came to modelling, issues arose with the size of the Corpus. There is a limitation in spaCy, it will only work with text files less than 100,000,000 characters long. I think that each book was a little over twice that size. So I needed to batch the process by breaking the corpus up into smaller sections. This was done with the split_segments function from the rainette package. The function only accepts a split based on number of sentences, so I arrived at a value of 100,000 sentences per document.

corpus <- list_of_files %>% 
        map_df(readtext, encoding='utf-8') %>%
        corpus() %>%
        rainette::split_segments(segment_size = 100000)

  Splitting...

  Done.

With the books read into file, the corpus created and the corpus split into sections, we now have 18 documents. We can proceed to entity modelling with the spaCy functions.

Unfortunately, we still have size issues, as passing the entire Corpus to be parsed is unaffected by the number of documents. So I needed to create a simple for loop to analyze each document one at a time and bind the results to a data table. Data tables are like data frames, but they have some unique notation and increased performance.

The Corpus is parsed with the spacy_parse function. Setting pos and lemma to false should reduce performance time as the function doesn’t need to return dependency POS tag set and lemmatized tokens. The POS tag set refers to the type of word such as Noun, while the lemmatized token is the base of a word, such as for the word “am” would be lemmatized to “be”. The parsing of the corpus takes a very long time.

df <- corpus[[1]] %>%
        spacy_parse(pos = FALSE, lemma = FALSE)

Found 'spacy_condaenv'. spacyr will use this environment

successfully initialized (spaCy Version: 3.1.3, language model: en_core_web_sm)

(python options: type = "condaenv", value = "spacy_condaenv")

for (i in 2:length(corpus)){
        temp <- corpus[[i]] %>%
        spacy_parse(pos = FALSE, lemma = FALSE) 
        
        df <- rbind(df,temp)}
rm(temp)

The parsing creates an object that acts very similarly as a data table. There is an entry for each word, which is more than what is required for this project. The original data table is preserved, in case we would like to reference a sentence in the corpus, and we create a filtered data table. The data table is filtering the tokens in the names list and by the identified entity, making sure it starts with person.

dfclean <- df %>% 
        filter(token %in% names,
               str_starts(entity, "PERSON")) 

Relationship modelling

The final step is to create a model that will connect people in the data table. I have decided to use a sentence windows that creates a connection when two names are mentioned within that window.

This is another very time-consuming tasks the requires two for loops. The first loop goes through all 30025 rows and its sentence id. A second for loop that excludes all rows already used in the first loop is used to compare a second sentence id. If the difference between the sentence ids is less than the windows, the tokens for these rows are added to an empty data table. If the difference is greater than the window size, we break the second for loop as all the sentence ids are incremental. It is not a very clean or smooth method, but it works.

window_size <- 5
related <- data.table("Person1" = character(), "Person2" = character())

for(i in 1: (nrow(dfclean)-1)){
        for(j in (i + 1):nrow(dfclean)){
                if((dfclean$sentence_id[j] - dfclean$sentence_id[i]) < window_size){
                        
                        related <- rbindlist(list(related, list(dfclean$token[i], dfclean$token[j])))
                }
                
                else{
                        break
                }
        }
}

The following is a sample of the data table we have created to build the relationships.

related %>% head()

   Person1 Person2
1: Shallan  Kabsal
2: Jezrien Jezrien
3: Jezrien Jezrien
4: Jezrien Jezrien
5: Jezrien Jezrien
6: Jezrien   Kalak

We can identify two issues with this sample. The first issue is when two of the same names are within the same window. We will have to filter out when ‘Person1’ is equal to ‘Person2’. The second issue is that we would actually like to aggregate the data. We would like a count of when two different names are in the same window. Both of these tasks are easy enough to solve using the built-in data table notation. For more information on data tables, please refer to my previous post on the topic.

relatedagg <- related[Person1 != Person2,.N,by = c("Person1", "Person2")]

relatedagg %>% head()

   Person1 Person2  N
1: Shallan  Kabsal  8
2: Jezrien   Kalak 10
3:   Kalak Jezrien  9
4:   Szeth Dalinar 98
5:   Szeth  Jasnah 11
6:   Szeth Elhokar  3

The final issue is for the relationships for ‘Person1’ and ‘Person2’ when their places are switched, but that will be dealt with in the next post.

Conclusion

With some hard-work, we were able to create an organized Corpus of all the current 4 Stormlight Archive books. We were able to split this Corpus into smaller sized documents, making them easier to manage. The spacyr library was then used to model entities within the Corpus, identifying the tokens that represent people. The next step was to clean up the results, keeping only the verified characters names as tokens. We then used a model to developed relationships using a window. A relationship was created whenever two character names were mentioned in the same window. We then filtered out characters relationships to themselves and aggregated the data. The clear next step is to actually build the graphs with the characters as nodes and their relationships as edges. But that is a post for another day.

Introduction

This project is inspired from a YouTube video created by Thu Vu about at data scraping project about the Witcher books series. Her project utilizes python and Selenium. I love the book series and I loved the project idea. I’ve also had it on my backlog to learn the Rvest library for a while, so it seems like a great opportunity to combine these two interests.

Rather than completing the project on the Witcher series, I thought it would be interesting to explore another book series that I love in the Stormlight Archive by Brandon Sanderson. If you are not familiar with the series, it is an epic fantasy story sprawling over four main books at the time of the publishing of this post. Sanderson is a fantastic author and I feel that the Stormlight Archive is his best work.

For this project, I will scrap the Coppermind website for all the character names in the series. The Coppermind is a fan made Wiki site that covers the work of Brandon Sanderson. After retrieving all the character names, I will create a graph outlining the relationships between each character. This work will be done in a future post, so please look forward to it.

Inializaton

The first step is to download the Rvest library. This is done with the following code.

install.packages("rvest")

The rvest package is also installed if you have the tidyverse package installed. Loading the tidyverse package however will not load the rvest package, so they both need to be loaded separately.

library("tidyverse")
library("rvest")

Datascraping

To start the data scraping exercise, we need to save the URL of the website we would like to scrape. This is the URL for the character page for the Stormlight Archives series.

site <- read_html("https://coppermind.net/wiki/Category:Rosharans")

While on the website in your own browser, you right-click on the specific element you’re interested in scrapping and select inspect. This is at least the method used for Firefox, but it should be similar to other browsers.

From here, you have to do a little digging and a little experimentation to determine which HTML elements are important for the character list. It is pretty useful to have a strong understanding of HTML at this point. From my experimentation, I found that the list was contained within a div with the class “mw-category-group”. A div is a generic divider tag in HTML and can represent many things. I selected the elements with the following code:

names <- site %>% 
        html_elements("div.mw-category-group")

You use the html_elements command to select the all the elements for a specific HTML tag you pass. The addition of the “.mw-category-group”, specifies the selection to only divs with the specific class. The class is an attribute of the HTML tags, used to identify and group HTML elements together. I have found that this notation is the best way to filter elements.

Within the div elements, there is a further sub-structure for an element in the character list. The characters are contained within an <li> tag as a list item and as an <a> tag as a hyperlink within that list item. We can explore further into the HTML structure by selecting these elements. After the final structure is selected, we can use the html_attr function to return an attribute of the selected elements. The ‘title’ attribute stores the character name in the HTML. We could also the html_text2 function to return the text of the hyperlink, but I’ve found that the ‘title’ attribute is better structured.

names <- names %>%
        html_elements("li") %>%
        html_elements("a") %>%
        html_attr("title")

Data Cleaning

We can start exploring the results of the scrapping

print(head(names))

[1] "Category:Aimians"    "Category:Alethi"     "Category:Azish"     
[4] "Category:Emuli"      "Category:Herdazians" "Category:Iriali"    

Oops, the program has captured an additional list that precedes the character list. Through my testing, I have not found a way to distinguish between the two lists from the HTML structure. Thankfully, we can rely on Regular Expressions to complete the job. The unwanted list items all start with “Category:”, so with a single expression of the str_starts from the stringr package we can remove these elements.

names <- names[!str_starts(names, "Category:")]

The list still requires some additional work, as there are “()” used throughout the list to give additional context. These “()” will not appear in the text, so we need to remove them with a second Regular Expression. Although it is not clear to me, the “(” needs to be double escaped with two “\” rather than just one.

names <- str_remove_all(names," \\(.*\\)")

print(head(names, 10))

 [1] "Abaray"        "Abiajan"       "Abrial"        "Abrobadar"    
 [5] "Abronai"       "Abry"          "Acis"          "Adin"         
 [9] "Adis"          "Adolin Kholin"

We can see that the scrapped data is in much better condition. There is still additional work we can do, as the names will sometimes include first and last names. The last names are not particularly important, so we can drop them altogether.

names <- str_remove_all(names," .*") 

print(head(names, 10))

 [1] "Abaray"    "Abiajan"   "Abrial"    "Abrobadar" "Abronai"   "Abry"     
 [7] "Acis"      "Adin"      "Adis"      "Adolin"   

names <- names[!names %in% c("Word","She", "User:Thurin")]

Now we can see that the final list is in condition that we can use to better explore the relationships.

Conclusion

We have scraped the Stormlight Archive character wiki website with the rvest package. We loaded the website with read_html function. Furthermore, we were then able to sort through the different HTML elements with the html_elements to find where the character list is stored. We then obtained the actual names with the html_attr function. The data collected still contained some unwanted data. We were able to remove an additional list, data in parentheses and the last names of all characters. We can now move forward with scrapping the books to identify the strength of relationships between each character.

Description

This is the second part for the creation of a text prediction Shiny Application. From the previous post, we have developed and Corpus of text to start creating text prediction applications.

We have also explored the corpus, looking at the frequency of words in the vocabulary. It is now time to start to develop ngram models.

N-gram models

A ngram is a continuous sequence of tokens, where the order is determined by how many tokens are in the sequence. For our purpose, a token is created for each word in a sentence. Other tokens can be created, such as sentence in a paragraph or letters in a word. It really depends on your application needs.

A line of text can be broken down into ngrams in many ways. For example, the following text:

“The quick brown fox”

can be broken down to the following unigrams:

(“the”)(“quick”)(“brown”)(“fox”)

or to the following bigrams:

(“the quick”)(“quick brown”)(“brown fox”)

or to the following trigrams:

(“the quick brown”)(“quick brown fox”)

or to the single tetragram:

(“the quick brown fox”)

The process for creating tokens from text, tokenization, drops the text to lower case and removes all punctuation. For this application, I would recommend the unnest_tokens function from the tidytext package.

Ngrams can be used for predictive text by reserving the last word in the ngram as the predicted word.

Models

truetext <- reactive({
        truetext <- input$text %>%
                tibble(text=.) %>%
                unnest_tokens(word, text, token="ngrams", n=1)
        
        truetext[!truetext$word %in% voc$word,] <- "unk"
        truetext})
        
        maxuse <- reactive({
                min(nrow(truetext()) + 1,maxn)
                })

df <- ngrams$three %>%
        filter(word1 =="what", word2 == "is") %>%
        select(word3, n) %>% 
        right_join(voc, by = c("word3" = "word")) %>%
        mutate(bin = substr(word3,1,1)) %>% 
        group_by(bin)

df$n[is.na(df$n)] <- 0

df %>%
        ggplot(aes(x = bin, y = n)) +
        geom_bar(stat = "identity")

df <- ngrams$three %>%
        filter(word1 =="what", word2 == "is") %>%
        select(word3, n) %>% 
        right_join(voc, by = c("word3" = "word")) %>%
        mutate(bin = substr(word3,1,1)) %>% 
        group_by(bin)

df$n[is.na(df$n)] <- 1

df %>%
        ggplot(aes(x = bin, y = n)) +
        geom_bar(stat = "identity")

df <- ngrams$three %>%
        filter(word1 =="what", word2 == "is") %>%
        select(word3, n) %>% 
        right_join(voc, by = c("word3" = "word"))

df$n[is.na(df$n)] <- 0

Nr <- count(df, n, name = "nn")


Nr %>%
        head()

# A tibble: 6 × 2
      n    nn
  <dbl> <int>
1     0 64330
2     2    51
3     3    28
4     4    13
5     5     2
6     6     7

total <- sum(Nr$nn*Nr$n)
total

[1] 1449

Nr <- Nr %>%
        arrange(n) %>% 
        mutate(c= 0) %>%
        mutate(sc = 0) %>%
        mutate(GT = 0)

#the probability for unseen matches is set to the next value probability
Nr$GT[Nr$n==0] <- Nr$nn[2]*Nr$n[2]/total

Zn <- Nr[-1,] %>% add_row(n=Nr$n[nrow(Nr)]+1)
Zr <- Nr[-1,] %>% lm(log(nn)~log(n), data=.) %>% predict(newdata=Zn)
Zr <- exp(Zr)

#creates the new adjusted counts
j <- 0 
for (i in 2:nrow(Nr)) {
        Nr$c[i] <-  (Nr$n[i]+1)*Nr$nn[i+1]/Nr$nn[i]
        Nr$c[i][is.na(Nr$c[i])] <- 0
        Nr$sc[i] <-  (Nr$n[i]+1)*Zr[i]/Zr[i-1]
        if(Nr$n[i+1]-Nr$n[i] > 1 | i == nrow(Nr)){
                j <- 1}
        Nr$GT[i] <-  Nr$c[i]*(1-j) + Nr$sc[i]*j
        }

#the specific prop from words with the same count
Nr$GT[Nr$GT < 0] <- Nr$nn[2]/total
Nr$GT <- Nr$GT/sum(Nr$GT)
Nr$GT2 <- Nr$GT/Nr$nn

df <- ngrams$three %>%
        filter(word1 =="what", word2 == "is") %>%
        select(word3, n) %>% 
        right_join(voc, by = c("word3" = "word")) %>%
        mutate(bin = substr(word3,1,1)) %>% 
        group_by(bin)

df$n[is.na(df$n)] <- 0

df %>%
        left_join(select(Nr,n,GT2), by = "n") %>%
        ggplot(aes(x = bin, y = GT2)) +
        geom_bar(stat = "identity")

Nr %>%
        select(c,sc) %>%
        mutate(diff = c-sc) %>%
        ggplot(aes(x=diff)) +
        geom_histogram()

discount <- 0.75
ADI <- df %>%
        ungroup() %>%
        select(word3, n) %>%
        mutate(ADI = (n - discount)/sum(n))
                
ADI$ADI[ADI$ADI < 0 ] <- 0

unigram.prop <- ngrams$one %>%
        mutate(prop = n / sum(n))

uni.wt <- 1 - sum(ADI$ADI)

ADI <- ADI %>% 
        add_column(uni = unigram.prop$prop*uni.wt) %>% 
        mutate(ADI = ADI + uni, .keep = "unused")

ADI %>%
        mutate(bin = substr(word3,1,1)) %>% 
        group_by(bin) %>%
        ggplot(aes(x = bin, y = ADI)) +
        geom_bar(stat = "identity")

KNS <- df %>%
        ungroup %>%
        select(word3, n) %>%
        mutate(KNS = (n - discount)/sum(n))

KNS$KNS[KNS$KNS < 0 ] <- 0
cont.wt <- 1 - sum(KNS$KNS)

cont.prop.func <- function(word, ngrams){
        out <- ngrams %>% 
                filter(.[,ncol(ngrams)-1] == word) %>%
                nrow() 
        out / nrow(ngrams)
}
cont.prop <- list()
cont.prop$one <- tibble(word=voc$word, prop = ngrams$one$n/sum(ngrams$one$n))
cont.prop$two <- tibble(word=voc$word, prop = map_dbl(word, cont.prop.func, ngrams=ngrams$two))
cont.prop$three <- tibble(word=voc$word, prop = map_dbl(word, cont.prop.func, ngrams=ngrams$three))
cont.prop$four <- tibble(word=voc$word, prop = map_dbl(word, cont.prop.func, ngrams=ngrams$four))
cont.prop$five <- tibble(word=voc$word, prop = map_dbl(word, cont.prop.func, ngrams=ngrams$five))
cont.prop$six <- tibble(word=voc$word, prop = map_dbl(word, cont.prop.func, ngrams=ngrams$six))
saveRDS(cont.prop, "cont.prop.rds")

KNS$KNS <- KNS$KNS + cont.prop$three$prop*cont.wt

KNS %>%
        mutate(bin = substr(word3,1,1)) %>% 
        group_by(bin) %>%
        ggplot(aes(x = bin, y = KNS)) +
        geom_bar(stat = "identity")

Shiny App

With all the models created, we can bundle it together in a single Shiny Application. This Shiny Application retrieves the user’s input and attempts to predict the next word. A table is generated to summarize the most highly predicted word. Since there are five different models, there are five different rows. A plot is generated for each model where the predicted words are in bins with other words with the same first letter.

Photo by Jaredd Craig on Unsplash

Structure

The basic structure of a list comprehension is pretty simple, you contain an expression and an iterable within a set of []. Depending on the type of brackets used, you can create a list, a generator, set or a dictionary.

[i for i in range(5)]
(i for i in range(5))
{i for i in range(5)}

{0, 1, 2, 3, 4}

It may appear from first impressions that a list comprehension is a simple one line for loop, but it is much more powerful than that.

[a for a in range(10) if a % 2 == 0]

[0, 2, 4, 6, 8]

[a if a % 2 == 0 else 0 for a in range(10)]

[0, 0, 2, 0, 4, 0, 6, 0, 8, 0]

a = ['0', '1', '2', '3', '4']
[int(x) for x in a]

[0, 1, 2, 3, 4]

a = range(5)
b = [5,10,15]
[x*y for x in a for y in b]

[0, 0, 0, 5, 10, 15, 10, 20, 30, 15, 30, 45, 20, 40, 60]

{char : num for num, char in enumerate(['a','b','c','d','e'])}

{'a': 0, 'b': 1, 'c': 2, 'd': 3, 'e': 4}

a = range(5)
[True for x in a]

[True, True, True, True, True]

[x for x in [b for b in range(20) if b %2 == 0] if x %3 == 0]

[0, 6, 12, 18]

Structure of an F string

The formatting of an F string starts with a f prior to quotations, whether they be single or double quotes. Any variable can then be included within a series of {}. This formatting can make it easier than turning values into strings and concatenating all strings into a single line of text. This is easily demonstrated with a large mix of values and strings.

x = 1/3
y = 1/6

print("The value is " + str(x) + " is greater than " + str(y))
print(f"The value is {x} is greater than {y}")

The value is 0.3333333333333333 is greater than 0.16666666666666666
The value is 0.3333333333333333 is greater than 0.16666666666666666

The values can then be formatted with : after the variable name. The number of digits prior and post the decimal can then be specified. The f is added after the decimal formatting to ensure the value is treated as a float.

print(f"The value is {x:.3f} is greater than {y:.2f}")

The value is 0.333 is greater than 0.17

The values passed are not specific to the number of digits, but the minimum number of spaces. This means you can ensure specific space aligned, such as for a table, by including these values.

z = [10000, 500, 10, 0.001, .1]
for i in z:
        print(f"the value is: {i:5}")

the value is: 10000
the value is:   500
the value is:    10
the value is: 0.001
the value is:   0.1

Additionally, we can add leading zeros by adding zero prior to the number of digits.

for i in z:
        print(f"the value is: {i:05}")

the value is: 10000
the value is: 00500
the value is: 00010
the value is: 0.001
the value is: 000.1

Alternative formatting

There are a few alternative methods for f strings. From my understanding, they are not as fast when it comes to performance. I don’t think that is of particular importance. If your script needs a high level of performance, than you probably don’t want many print statements.

print("The value is {} is greater than {}".format(1/3, 1/6))

The value is 0.3333333333333333 is greater than 0.16666666666666666

print("The value is %5.3f is greater than %5.3f" %(x,y))

The value is 0.333 is greater than 0.167

Conclusions

Whichever method you decide, it probably won’t make a huge difference. The important part is to understand is the actual formatting. F strings also seem to make it easier to understand the code, as the actual values are inline with the string and the formatting.

Photo by Sigmund on Unsplash

Deliberate Practice

Deliberate Practice can be summarized to the following points:

• Talent is not enough, and to become great at a task requires a lot of practice and repetition.

• Deliberate practice is hard-work, in order strengthen your skills through practice you need to be challenged. This means that repetition by itself will not develop skill. This also means you need to constantly increase the challenge of your practice as you become better at it.

• Focus plays a large role in deliberate practice. This connects to the previous point for the required challenge of practice. This can also tie into the principles of flow, as best described by Mihaly Csikszentmihalyi in his book “Flow: The Psychology of Optimal Experience”.

• Setting goals becomes a powerful motivator. With the completion of a goal, there is a release of endorphins, which cause a sense of satisfaction. Goals are can also be utilized to increase the difficulty of practice, making an otherwise easy task a challenge.

• Feedback is important. Feedback provides motivation by comparing current to previous performance.

Application for programmers

So, how can programmers incorporate the principles of deliberate practice? I’ve recently been recommended the site Hacker Rank and I can say it is fantastic. The Hacker Rank site provides a wide array of challenges for programmers of varying skills levels. There is a selection of different topics from Algorithms to Regular Expressions.

So how does Hacker Rank fit in with deliberate practice? Well, there certainly are a good level of challenges to work through. There a three different levels per topic, with challenges in the same difficulty having nothing in common.

The design of the site is quite simple, with very little to distract you from your challenge. There is the option for a dark theme if you are, like me, a person of sophistication. It also pretty easy to have the problem on one side of your screen with your programming on the other.

There are multiple built-in goals to work on. Certifications for each topic to test your current skill level and to advertise to potential employers. There a preparation kits for interviews with time frames between 1 and 12 weeks, which ever is most convenient for you.

There is immediate feedback from assignments, with automated program testing. A leader board is provided for the most competitive, who are interested in their global ranking. You also get feedback from the built-in IDE on your programming errors.

Site Criticisms

I do enjoy the site, but I still have some minor issues. To its credit, the site does support multiple programming languages, even different versions of the same programming language. This does, however, make it difficult to following along with the tutorials if they are done in a language that you are not familiar with.

Also, at this time, there doesn’t seem to be support to retake certification exams for some topics. I myself, had failed the R basic certification as I am more used to using the tidyverse package rather than base R.

I can’t really speak on behalf of the incorporation of potential employers in it. But that does seem like a very promising idea. I still think it is a great to tool for the programming community, and I will continue to utilize it for my personal skill development, as it can easily provide a source of deliberate practice.

Application Structure

My shiny dashboard will at heart be a shiny application. I will use the single App file structure rather than the separate UI/Server file structure. The R Studio site has a very good outline on the basic structure for a Shiny Dashboard.

MY application will then be broken down into the following structure:

• Data Collection
• Plot Creation
• UI
• Server

Before starting to write the Shiny App, the raw data needs to be cleaned and setup to increase application performance.

Data Preparation

The data is the same from the previous dashboard, which was the level of donations for Canadians to charities. The raw data can be found here.

The following code is used to clean up the raw data and is not included in the Shiny App. This code is just used to create an RDS file (compressed data file) that the Shiny App will more easily load.

library(tidyverse)
# Download the data and unzip
download.file("https://www150.statcan.gc.ca/n1/tbl/csv/45100007-eng.zip", "donordata.zip")
unzip("donordata.zip")

# Read the data into R
data = read_csv("45100007.csv")

# Clean up the data
data = data %>%
        filter(`Donation statistics (UOM)`=='Average annual donations') %>%
        filter(!GEO=="Canada") %>%
        filter(!Education == "All education levels") %>% rename(Province = GEO)
data$Province[data$Province == "Newfoundland and Labrador"] <- "Newfoundland"
data$Province[data$Province == "Prince Edward Island"] <- "P.E.I."

# Saved the clean data as an RDS file
saveRDS(data, 'data.rds')

Data Collection

With the data preparation completed, we can now start with writing the application. Shiny Application can be broken down to two parts, reactive and nonreactive. It is important to keep our calculations in the nonreactive part if their values do not change because it will be very demanding on system resources otherwise.

# Loading the libraries
library(shiny)
library(shinydashboard)
library(plotly)
library(tidyverse)
library(leaflet)
library(rgdal)
library(gt)

# Data is loaded into the shiny app from the previously generated RDS file
data = readRDS('data.rds')

# Summary data is created from the loaded data and saved as data2
data2 <- data %>%
        group_by(Province) %>%
        summarise(Donations = sum(VALUE))

# For regional information for mapping, the rgdal library is used.
library(rgdal)

# The following code will download a regional outlines for maps if the file doesn't exist on the system 
if (!file.exists("./src/ref/ne_50m_admin_1_states_provinces_lakes/ne_50m_admin_1_states_provinces_lakes.dbf")){
        download.file(file.path('https://www.naturalearthdata.com/http/',
                                'www.naturalearthdata.com/download/50m/cultural',
                                'ne_50m_admin_1_states_provinces_lakes.zip'), 
                      f <- tempfile())
        unzip(f, exdir = "./src/ref/ne_50m_admin_1_states_provinces_lakes")
        rm(f)
}

# The regional data is then loaded into R and some data is edited to make it more inline with the regional data
region <- readOGR("./src/ref/ne_50m_admin_1_states_provinces_lakes", 'ne_50m_admin_1_states_provinces_lakes', encoding='UTF-8', verbose = FALSE
                  )
data2$Province <- c("Alberta", "British Columbia", "Manitoba", "New Brunswick", "Newfoundland and Labrador", "Nova Scotia", "Ontario", "Prince Edward Island", "Québec", "Saskatchewan")

Plot Creation

Just as the data was collected in the nonreactive section, so should the plot creations. This doesn’t mean that the plots won’t be interactive, just that their designs will remain static.

bar_plot <- data %>%
        group_by(Province) %>%
        summarise(Donations = sum(VALUE)) %>%
        ggplot(aes(x = Province, y = Donations, fill = Province)) +
        geom_bar(stat = "identity", show.legend = FALSE) +
        theme(axis.text.x = element_text(angle = 90), legend.position='none')

# This call was added for illustration
bar_plot

edu_plot <- data %>%
        group_by(Education) %>%
        rename(Donations = VALUE) %>%
        ggplot(aes(y= Donations, x = Education, fill = Education)) +
        geom_boxplot() +
        theme(axis.title.x=element_blank(),
              axis.text.x=element_blank(),
              axis.ticks.x=element_blank())

# This call was added for illustration
edu_plot

pie_plot <- data %>%
        group_by(Province) %>%
        summarise(Donations = sum(VALUE)) %>%
        ggplot(aes(x = '', y = Donations, fill = Province)) +
        geom_bar(stat = "identity", width = 1) +
        coord_polar("y", start = 0) +
        theme_void()

# This call was added for illustration
pie_plot

map_leaf <- leaflet() %>% 
        addTiles() %>% 
        setView(-74.09, 45.7,  zoom = 2) %>% 
        addPolygons(data = subset(region, name %in% data2$Province), color = "#444444", opacity = 1.0, fillOpacity = 0.75,
                    fillColor = ~colorQuantile("Greens", data2$Donations)(data2$Donations),
                    weight = 1)

# This call was added for illustration
map_leaf

gt_table <- data2 %>% 
        pivot_wider(names_from = Province, values_from = Donations) %>%
        gt()

# This call was added for illustration
gt_table

UI

The following code creates the UI interface that the user interacts with. For those familiar to Shiny Apps, the structure will be similar. The actual functions used will be different, but usually their names will just have the ‘dashboard’ prefix. For those not familiar with Shiny Apps, the UI is created by a nesting of a series of functions. The notation can get a little difficult as you need to remember to use ‘,’ when passing multiple functions. Though not used in this dashboard, UI interfaces can be used to control elements of the display. These controls create variables that need to be saved as input$"variable_name". This allows the transfer of information from the UI to the server side. Likewise, plots and figures from the server side need to be saved as output$"variable_name". When an output variable is referenced in the UI, such as the plotOutput function for displaying plots, they are referenced as “variable_name” in quotes.

Again, the syntax can be difficult to learn and use, but the R Studio post on Shiny Dashboards provide the basic skeleton to learn from. My biggest issue is keeping track of all the ‘)’ and ‘,’ used.

# The main page structure is the dashboardPage
ui <- dashboardPage(
        
# The header is simple the header for the dashboard
        dashboardHeader(title = "Donations in Canada Dashboard"),

# The sidebar functions adds the controls for flipping through dashboard pages        
        dashboardSidebar(
                
# Each menu item is a link that will apear on the sidebar
# There are a bunch of quality icons that the icon function can return 
                menuItem("Dashboard", tabName = "gauge", icon = icon("dashboard")),
                menuItem("Education", tabName = "education", icon = icon("school")),
                menuItem("Map", tabName = "map", icon = icon("map"))),
        
# The next section includes the body or the main section of the dashboard
        dashboardBody(
# In the main body, each of the tabs in the side bar need to be referenced. The tabItems function is the container for all the tabs
                tabItems(
                        
# Each previous mentioned tab, will have a separate tabItem.
                        tabItem(tabName = "dashboard",

# The fluidRow function is used to align the elements on the page. Through trial and error, I have found this to be the element for changing the size of plots.
                                fluidRow(
                                        
# A box is drawn around each plot. Each plot/figure/element is referenced here with the plotOutput function or a likewise function. In the server side, I will transform the ggplots into plotly plots to create interactive plots so the plotlyOutput function is used. 
                box(plotlyOutput("plot1"), width = 6),
                
# The pie plot is a regular ggplot object, so it can be passed to the plotOutput function
                box(plotOutput("pie"), width = 6))),
                
                tabItem(tabName = "education",
                        fluidRow(
                                box(plotlyOutput("plot3"), width = 12))),
                
# For the map tab, the leafletOutput function is used to display the leaflet map created in the server side. 
                tabItem(tabName = "map", 
                        fluidRow(
                                box(leafletOutput("plot2"), width = 8)
                                ),
                        
# On a second row, under the map, a GT table requires the gt_output function
                        fluidRow(
                                box(gt_output("table"), width = 12)
                        ),
                        )
                )
        )
)

Server

The server is the second major function in the Shiny App. Rather than using the output series of functions, it requires the paired version of the render function. Any controls from the UI would need to be passed as part of the input variable.

# The same structure for any other Shiny App
server <- function(input, output) {

# Any data to pass back to the UI, such as figures or plots, will need to be saved as part of the output. Anything within the {} brackets becomes an interactive element. If they are not present, the plot will not change.
        output$plot1 <- renderPlotly({
                
# The ggplot for the barplot is converted to an interactive plotly plot.
                        ggplotly(bar_plot)
                })

        output$plot2 <- renderLeaflet({map_leaf})
        
        output$plot3 <- renderPlotly({ggplotly(edu_plot)})
        
# The GT table requires the render_gt function which pairs with the gt_output function in the UI.
        output$table <- render_gt({gt_table})
        
        output$pie <- renderPlot({pie_plot})
}

Finalized Dashboard

The finalized dashboard looks pretty good. The sidebar adds a good level of control to the user and makes the data better organized. I also enjoy the default styling, but I’m sure it can be changed.

When compared to the dashboard created with flexdashboard, it is much nicer looking. The flex dashboard, however, has the advantage of being much easier to create with r markdown. It is also much more flexible as it produces an HTML file which is easy to manage. The shiny dashboard needs to be hosted, which makes it slightly clunky.

Speaking of the clunky nature of Shiny Apps, you also need to be conscious of what needs to be reactive and what doesn’t. It is very easy to slow down a Shiny Dashboard with a bunch of unnecessary calculations. The writing of the application itself is also quite clunky. It doesn’t compare to the ease of writing in R markdown.

So which dashboard do I prefer? Well I think for a quick solution, flex dashboard is great but the amount of control and styling of a shiny dashboard is just better. I do think that there is a place for both, but if I had to choose, I would go for a Shiny Dashboard.

Introduction

Data Tables are a potential replacement for the common dataframe. It seeks to perform that same role but with improved performance. I would like to see the speed comparison between Data Frames, Data Tables and Tibbles. I will use the microbenchmark package to perform the actual benchmarking.

library(tidyverse)
library(data.table)
library(microbenchmark)
library(farff)

For the benchmark, I will use the ‘credit-g’ dataset, which can be found on the open ml website. I’m pretty sure the last open ml dataset I used was a csv file, but they seem to have moved to a ARFF format. I will need to use the farff package to load the data.

df <- farff::readARFF('dataset_31_credit-g.arff')
dt <- setDT(df)
ti <- tibble(df)

Syntax

The syntax for Data Tables is a little different:

DT[i,j,by]

In this manner, a data table can be subset by i, to calculate j when grouped with a by. Along with the special syntax, there are some common functions that add some additional simplification.

.()

The ‘.()’ function can be used as a placeholder for ‘list()’. The list function is useful for subsetting.

Grouped Aggregate

Aggregating data in Data Tables is simple by using the j and by parameters in the syntax. Again, multiple functions or even multiple groupings can be passed with the ‘.()’ function. For this comparison, we will look at the performance of finding the average age of the credit holders grouped by the class or credit rating.

group <- microbenchmark(Data_Frame = df %>% 
                                 group_by(class) %>%
                       summarise(avg = mean(age)),
               Data_Table = dt[,.(avg = mean(age)), by = class],
               Tibble = ti %>% 
                       group_by(class) %>%
                       summarise(avg = mean(age)))
print(group)

Unit: microseconds
       expr    min     lq     mean  median      uq     max neval
 Data_Frame 5406.5 5564.6 6138.870 5904.40 6306.95 17135.6   100
 Data_Table  623.6  766.8  941.930  885.95  958.05  7471.7   100
     Tibble 5602.8 5783.1 6586.445 6103.15 6902.65 14356.9   100

Taking counts

Another function of interest is the ‘.N’ function. This function will return the count of rows. The test looks are the number of people with over 5000 in credit and younger than 35.

counts <- microbenchmark(Data_Frame = df %>% 
                                 filter(credit_amount > 5000, age <35) %>%
                       nrow(),
               Data_Table = dt[credit_amount > 5000 & age < 35, .N ,],
               Tibble = ti %>% 
                       filter(credit_amount > 5000, age <35) %>%
                       nrow())
print(counts)

Unit: microseconds
       expr    min      lq      mean  median      uq     max neval
 Data_Frame 8404.1 8584.90  9576.638 8927.45 9867.95 17254.5   100
 Data_Table  276.7  309.50   429.832  451.80  466.35  1045.6   100
     Tibble 8945.1 9204.75 10206.312 9405.45 9942.45 27081.9   100

Creating new columns

Data Tables also contain a very simple syntax for creating a new column with ‘:=’. I compare this to the tidyverse mutate function. Using the base R to create a column is still the fastest method, taking about half the time of the Data Table method.

new <- microbenchmark(Data_Frame = df %>% mutate(property = paste(property_magnitude, housing)),
               Data_Table = dt[,property := paste(property_magnitude, housing)],
               Tibble = ti %>% mutate(property = paste(property_magnitude, housing)))
print(new)

Unit: microseconds
       expr    min      lq     mean median      uq    max neval
 Data_Frame 2071.3 2148.55 2396.203 2259.6 2586.40 3386.1   100
 Data_Table  505.7  580.00  657.802  656.6  694.35  996.0   100
     Tibble 2585.3 2754.85 3187.554 2922.9 3331.70 8879.8   100

Chaining Data Tables

Another point of exploration is that Data Tables can be chained together to create more complicated structures

dt[credit_amount > 1000, .(age = mean(age)),by = .(purpose, class)][class == "good" & age < mean(age)]

               purpose class      age
1:            radio/tv  good 35.44865
2: furniture/equipment  good 33.21930
3:            used car  good 36.91860
4:            business  good 34.50000
5:  domestic appliance  good 35.50000
6:          retraining  good 34.00000

I don’t think this is the most useful feature, as you can already create some very complicated transformation with a single call. Chaining also makes it more difficult to understand.

Conclusions

It is clear that there are significant performance improvements when using Data Tables versus Data Frames (an average decrease of time by -85%). There are also insignificant differences between Data Frames and Tibbles. Also, the syntax for Data Tables is fairly simple and straight forward and yet extremely powerful.

So, to answer the most important question, should you change to Data Tables from Data Frames? Probably, they present a significant performance gain and their structure is very flexible.

Photo by Tyler Clemmensen on Unsplash

Top CRAN downloads

With the following code, we can get the most popular packages from CRAN. The CRAN directory doesn’t represent all R packages, but a good amount of them.

library(tidyverse)
library(cranlogs)
top100 <- cran_top_downloads(when = 'last-month', count = 100)
top100 %>% head()

  rank  package   count       from         to
1    1  ggplot2 2502873 2022-07-17 2022-08-15
2    2    rlang 2039913 2022-07-17 2022-08-15
3    3 devtools 1844834 2022-07-17 2022-08-15
4    4       sf 1756905 2022-07-17 2022-08-15
5    5      cli 1672799 2022-07-17 2022-08-15
6    6     glue 1624788 2022-07-17 2022-08-15

From this list, we can see that the tidyverse represents a large amount of the top downloads with ggplot2, rlang and dplyr. The list includes the sf package for geospacial data, the glue package for string manipulation and the cli package which is used to create a command line interface for packages. Most of these packages I already have a good understanding of, so I need to narrow down the search.

Packages installed

You can get a list of your installed packages with the installed_packages function. You can then filter the top 100 list and remove anything you already have installed to find new packages.

mine <- installed.packages() %>%
        data.frame() %>%
        select(Package)
new <- top100 %>%
        filter(!package %in% mine$Package)
new

   rank     package   count       from         to
1     3    devtools 1844834 2022-07-17 2022-08-15
2     7        ragg 1589393 2022-07-17 2022-08-15
3     8 textshaping 1583922 2022-07-17 2022-08-15
4    10       rgeos 1524077 2022-07-17 2022-08-15
5    11         rgl 1499311 2022-07-17 2022-08-15
6    18     pkgdown 1162782 2022-07-17 2022-08-15
7    19  enrichwith 1158675 2022-07-17 2022-08-15
8    20      brglm2 1154128 2022-07-17 2022-08-15
9    31         zoo  873095 2022-07-17 2022-08-15
10   50       Hmisc  706994 2022-07-17 2022-08-15
11   60     rstatix  648757 2022-07-17 2022-08-15
12   62      nloptr  637641 2022-07-17 2022-08-15
13   63        lme4  623554 2022-07-17 2022-08-15
14   68    corrplot  601404 2022-07-17 2022-08-15
15   72       rJava  577002 2022-07-17 2022-08-15
16   75      ggpubr  544839 2022-07-17 2022-08-15
17   88     cowplot  493972 2022-07-17 2022-08-15
18   94         car  461659 2022-07-17 2022-08-15

From some quick research, I have found the following about the new packages:

• ragg - a 2D library as an alternative to the RStudio default
• rgl - functions for 3D interactive graphics
• rgeos - a geometry package, but is currently planned to be retired at the end of 2023 for the sf package
• zoo - a library to deal with time series
• pkgdown - a library fOR building a blog website, I use blogdown
• nloptr - a library for solving non-linear optimization problems
• Hmisc - an assortment of different data analysis tools
• lme4 - for fitting linear and generalized linear mixed-effects models
• RcppEigen - integration of the eigen library in R for linear algebra

Take-away

Hopefully your take-way is a simple method to explore R library that you have never heard about. I know that a few of the libraries seem interesting and worth further exploring.

While we are at it, might as well find the daily values for the new packages and plot them for the last month.

new$package %>%
        cran_downloads(when = "last-month") %>%
        ggplot(aes(x = date, y = count, color = package)) +
        geom_line()

Initialization

For the demonstration, I decided to use a data set on faults found in steel plates from the OpenML website. The data set includes 27 features with 7 binary predictors.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

df = pd.read_csv('https://www.openml.org/data/get_csv/1592296/php9xWOpn')

predictors = ['V28', 'V29', 'V30', 'V31', 'V32', 'V33', 'Class']
df['Class'] -= 1 

Since there are multiple binary predictors, I needed to create a single class variable to represent each class. The Class variable doesn’t currently represent this, it represents all faults that don’t fit in the categories of V28 to V33. The single variable class was created with the np.argmax function which returns the index of the highest value between all the predictors.

y = np.argmax(df[predictors].values, axis =1)
X = df.drop(predictors, axis = 1)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

Modelling

This is the most basic model that I would like to evaluate. I’ve used the GridSearch function, so all combinations of parameters are tested. The only parameter I wanted to examine was the size of the hidden layers. Each hidden layer provided is a tuple, where each number represents the number of nodes in a singled layer. Multiple numbers represent additional layers.

from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import GridSearchCV

parameters = {'hidden_layer_sizes':[(1),(100), (100,100), (100,100,100), 
(100,100,100,100), 
(100,100,100,100,100), 
(100,100,100,100,100,100), 
(100,100,100,100,100,100,100),
(100,100,100,100,100,100,100,100),
(100,100,100,100,100,100,100,100,100),
(100,100,100,100,100,100,100,100,100,100)]}
model = MLPClassifier(random_state = 1,max_iter = 10000, 
solver = 'adam', learning_rate = 'adaptive')

grid = GridSearchCV(estimator = model, param_grid = parameters)
grid.fit(X_train, y_train)
print(grid.best_score_)

0.4054982817869416

The performance of the best model in the grid is not impressive. It took me awhile to realize that I had forgotten to scale the features. I included this error to show the importance of scaling on model performance.

Feature Scaling

The features are simply scaled with the StandardScaler function. The same model is used on the scaled features.

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
scaler = sc.fit(X_train)
X_train_sc = scaler.transform(X_train)
X_test_sc = scaler.transform(X_test)

parameters = {'hidden_layer_sizes':[(1),(100), (100,100), (100,100,100), 
(100,100,100,100), 
(100,100,100,100,100), 
(100,100,100,100,100,100), 
(100,100,100,100,100,100,100),
(100,100,100,100,100,100,100,100),
(100,100,100,100,100,100,100,100,100),
(100,100,100,100,100,100,100,100,100,100)]}
model = MLPClassifier(random_state = 1,max_iter = 10000, 
solver = 'adam', learning_rate = 'adaptive')

grid = GridSearchCV(estimator = model, param_grid = parameters, cv=3)
grid.fit(X_train_sc, y_train)
grid.best_score_

0.7553264604810996

The performance of the scaled model is much more impressive. After the GridSearch function finds the parameters for the best model, it retrains the model on the entire dataset. This is because the function utilize cross validation, so some data was withheld for comparing the different models on test data.

Conclusion

With our model constructed, we can now test its performance on the original test set. It is important to remember to use the scaled test features, as that is what the model is expecting.

grid.score(X_test_sc, y_test)

0.7304526748971193

The results are pretty satisfactory. A decent level of accuracy without a lot of complicated code. Default values were used, whenever they were appropriate. Additional steps could be taken, but this remains a good foundation for future exploratory analysis.

Photo by Alina Grubnyak on Unsplash

Initialization

The Crosstalk package is available on CRAN and is loaded along with other important packages for this demonstration.

install.packages("crosstalk")
library(crosstalk)
library(tidyverse)
library(flexdashboard)
library(plotly)

I have decided to use a Toronto Open dataset about city audits for apartment buildings. I limited the features to only the ones that I feel will be interesting to look at. More information about the data set can be found here.

download.file("https://ckan0.cf.opendata.inter.prod-toronto.ca/dataset/4ef82789-e038-44ef-a478-a8f3590c3eb1/resource/979fb513-5186-41e9-bb23-7b5cc6b89915/download/Apartment%20Building%20Evaluation.csv", "data.csv")
df <- read_csv("data.csv") %>%
        select(lng = LONGITUDE, 
               lat = LATITUDE, 
               SCORE, 
               YEAR_BUILT, 
               SITE_ADDRESS, 
               PROPERTY_TYPE) %>% 
        slice_sample(n = 200)

The key to the crosstalk library is the SharedData functions. An object is created when a Data Frame is passed to the SharedData$new function. This is what enables communication between plots.

shared_df <- SharedData$new(df)

Dashboard Creation

The dashboard is created pretty much as previous mentioned in my dashboard post, with the exception that the shared Data Frame object is passed rather than the Data Frame.

The dashboard can include filters that are very similar to the Shiny Apt filters, with the filter_* family of functions.

filter_slider("Score", "SCORE", shared_df, ~SCORE, round = TRUE)
filter_checkbox("Property Type", "PROPERTY_TYPE", shared_df, ~PROPERTY_TYPE, inline = TRUE)

Conclusion

The Crosstalk package does add some significant connectivity to Flex Dashboards. It is relatively simple to use with some basic functions. It does have the issue of not working with aggregating data. The utility of finding the mean value of a selection is something Tableu and PowerBI are still superior at. I think that it is still a welcome improvement.

Final Dashboard

Photo by Jason Goodmanon Unsplash