Understanding Invoices with Document AI

At AppFolio, we specialize in helping Property Management Companies to automate their most repetitive, time-consuming, and tedious processes. Accounting-related tasks often check all of these boxes – and one of them is to maintain an accurate system of record for accounts payable and accounts receivable. This is a critical task due to strict federal and state regulations, and to ensure it is carried out accurately our customers use AppFolio’s Property Management (APM) software to enter invoices. However, these bills must be manually entered –  a time-consuming and error-prone process.

To help, we created Smart Bill Entry, a tool powered by state-of-the-art Machine Learning (ML) based models and our Document AI technology. Smart Bill Entry automatically extracts the most important information from the multitude of invoices a Property Manager receives in a given month - and in 2022 alone, we processed over 10 Million invoices. Here we will explore the tools enabling this technology and explain how it works behind the scenes.

Smart Bill Entry

Smart Bill Entry enables Property Managers to upload invoices in PDF format to APM directly via drag and drop, or by forwarding an email to a specific inbox. Once the invoice is uploaded, our ML models automatically extract key information, such as the amount, vendor name, the property’s address, and the invoice number associated with the bill.

We know Property Managers place a heavy emphasis on accuracy to ensure that bills get paid correctly. Based on our estimations, we have found that when manually entering invoices humans are roughly 95-99% accurate, depending on the field. Our model is designed to mirror this accuracy, and we compute calibrated confidence scores for each field to get our models to a state where they are as close to human accuracy as possible.

The model outputs a number between 0 and 1, where a score closer to 1 indicates that the model is confident that the class is correct. However, ML models tend to be over- or under-confident in their predictions, meaning that a given score doesn’t correspond to the probability that the prediction is correct - as demonstrated by the diagram below.

ML models tend to be over or underconfident, such that their confidence scores don’t exactly correspond to the probability that the prediction is correct. we fit additional calibration models To account for this, such that we can make consistent decisions of whether or not to show predictions. Image credit “On Calibration of Modern Neural Networks”. 

Calibration is the process of transforming the confidence score predicted by the model into a reliable probability that can be used to make decisions, like showing or not showing the prediction to the user. We do this as an added layer of precision. If our calibrated model is returning a confidence score of 0.95 for the “Bill Amount” field, we can expect that the model will be correct 95 times out of 100 predictions.

Document AI

The technology driving Smart Bill Entry’s ability to extract information from PDFs is Document AI. Document AI is a set of Machine Learning tools that are specifically designed for extracting information from a PDF, independent of the layout. The input data used to train our Document AI models includes two types of information: 

1. The Optical Character Recognition (OCR) information of the documents. This is the typed or written text on a document converted into machine-encoded text. Since we are only interested in fields that are critical to the invoice, we use datasets that zero in on this information.

2. The bounding box coordinates and content of the target fields we are looking to extract from the documents. We derive labels from these coordinates. To get the labels (e.g. “Vendor Name”, “Property Name”, etc.) for the content of each bounding box, human supervision is necessary - and the operators who assess the content of the invoices assign the appropriate label.

The image below is a visual representation of an invoice, the OCR information identified by the model for that invoice, and what labels would be generated for each object that is present on the invoice. 

Original Invoice vs OCR Information vs Labels

After collecting the OCR and bounding box dataset, we then apply different techniques that combine Computer Vision, Natural Language Processing, and Tabular Features. In the next section, we’ll discuss two of our approaches to extracting the above information, depending on our customer’s specific needs.  

Traditional vs Deep Learning

Smart Bill Entry combines two approaches to extracting information from invoices: traditional Machine Learning solutions and advanced Deep Learning solutions. For example, when extracting the ”Property Address” and the “Vendor Name” fields, we are using tree-based  models customized for each Property Manager. When we extract generic fields, such as the “Amount” and “Invoice Number” we use powerful DL models that can take advantage of layout and text using state-of-the-art Transformer architectures.

Traditional Machine Learning 

We extract the “Property Address” and “Vendor Name” fields from an invoice using traditional ML models. The input data for training the model is a simple Bag-of-words representation of the invoices, together with other engineered features relating to the layout. After extensive benchmarking we landed on a multi-class Random Forest Classifier as our base estimator. It fits several decision tree classifiers on various sub-samples of the dataset and uses averaging to improve predictive accuracy and prevent over-fitting.

Because AppFolio is a Property Management software used by thousands of companies (each with a large number of vendors and properties in their database), training only one multi-class Random Forest Classifier for all the vendors and properties in our database is a very challenging feat due to the high cardinality in the target variable and the challenge of deduplicating entities.

We train separate Random Forest ClassifierS for fields that have a unique set of target classes for each customer.

To tackle this, we decided to train one model per Property Management company. Not surprisingly, the drawback of this approach is that it requires a comprehensive and robust ML infrastructure to efficiently maintain and deploy thousands of models to production, as well as a short cold start phase to learn how to correctly map to specific vendor and property entities. It also means that each individual model needs to be relatively light weight and fast to train.
At AppFolio, we combined in-house solutions with third-party MLOps solutions, to create a state-of-the-art ML infrastructure that helps every team quickly train, deploy and monitor ML models at scale in production. We will talk more about our infrastructure in future posts in this blog.   

Deep Learning

For fields that have well defined labels across all customers such as the “Amount” and “Invoice Number”, we opted for a solution that implements a single Deep Learning model across all Property Management companies as opposed to one model per company. This approach generalizes well to unseen layouts and eliminates cold start issues.

Due to its very high learning capacity we were able to leverage almost all available training data in our database - which lends to our solution producing much more accurate predictions than traditional ML models. We also benchmarked against off the shelf Document AI solutions and found that our models significantly outperform them when evaluated on our holdout data.

Deep Learning Architecture

When deciding which DL architecture to implement for our model, we tried numerous approaches including: 

• Computer Vision models that use the image of an invoice as their input and output a bounding box for each class

• Natural Language Processing models that start from the OCR and classify each bounding box according to one of the given classes, and

• Multimodal models that use as input both the OCR and the image of the invoice. 

We implemented our models in such a way that we can exchange the architecture without modifying the input data and output format. In the image below, we show how different architectures (the yellow boxes in the diagram) can be exchanged without affecting the first and last layers of the model. This gives us the flexibility and agility to test different solutions and optimize our metrics. 

Keeping the input and output fixed we can easily switch between Deep learning architectures to optimize metrics such as accuracy, training and inference time. This also gives us the flexibility to quickly try out new approaches as the field advances.

When considering which architecture to deploy to production, we chose the solution that best balances accuracy with training time and inference speed. We used two processes to evaluate the performance of each model:

1. Offline Evaluation
After training a new model, we compare its performance against a frozen and static dataset that we use as a benchmark.

2. Online Evaluation
After training a new model, we deploy it in shadow-mode, where we do not show the predictions to the user, but rather just record them and compare metrics. 

Stay tuned for a future blog post where we will discuss the details of our ML infrastructure, and how we train, evaluate, deploy and monitor each model!

Authors and contributors: Ezequiel Esposito, Ari Polakof, Christfried Focke, Tony Froccaro

Lisa: Conversational AI

Inspired by PETER STEINER/The New Yorker magazine (1993) “On the internet, nobody knows you’re a dog.” Generated with DALL-E 2 prompt “Dog sitting on a chair in front of a computer with one paw on the keyboard. Comic style, black and white.”

In our last post we discussed the first step in the leasing process driven by Lisa, the Inquiry Parser. Once a message from an Internet Listing Service has been parsed, Lisa’s conversational AI is ready to chat with the prospective resident, either via text or email (we politely decline phone calls 🙂).

Lisa’s “Galaxy Brain”

The primary driver behind Lisa’s conversational AI is what we refer to colloquially as Galaxy Brain or Galaxy (the evolution, and constant intelligence gathering of Brain). Galaxy’s task is framed as multi-label text classification, and it works by converting the conversation into a structured response. We then use this response in Lisa’s logic layer to drive the conversation forward.

The structured response, pictured below, is a set of labels that are accompanied by confidence scores. The labels included in our model are:

• Intents - Tasks prospective residents want to accomplish

  • “This Thursday works for me”, “Can we do Friday instead?”, “I can no longer make the showing”
    (e.g. accept or counteroffer, reschedule, cancel)

• Categorical slot values - A piece of information that can be categorized

  •  (e.g. “I’m looking for a 1 bedroom”, “Can I do a virtual showing?”)

• Requested slots - A piece of information the prospective residents requests from us

  • (e.g. “What's the rent?”, “Do you accept Section 8?”)

• Acknowledgements - Cordial responses

  • (e.g. “You’re welcome!”, “Thank you for your time.”)

• Miscellaneous labels - Actions that change the back-end behavior of Lisa

  • (e.g. Mark the thread as spam, have the thread skip the operators’ inbox)

Galaxy Input text (top blue blob) and output response (bottom blue blob). The input text includes current inbound message (red) and conversation history (blue), and the output response includes confidence scores for each label.

The confidence score is a value between 0 and 1 that represents the likelihood that the output of the model is correct, with 1 being the highest. In this instance, because the confidence in SET_TOUR_TYPE_VIRTUAL is high, we would first mark the prospective resident’s preference for virtual tours, and then offer to schedule them a virtual tour. If this score were low, it may be handed off to an operator for review.

While highly accurate, a common problem with deep learning models is that they tend to be overconfident in their predictions. This means that they output a very high (or low) confidence score even if there is high uncertainty associated with the accuracy of the prediction. To adjust for this our model is fit with a set of calibration models, one per label, to map the confidence scores in such a way that they correspond more closely to the probability that the prediction is correct.

For non-categorical slot values, such as names, we use a separate Seq2Seq model similar to Lisa's Inquiry Parser.

NLP models transform the conversation history into a structured conversation state. A logic layer combines the conversation state with information from a Knowledge Base (KB) to compute the next action or response.

How Lisa Responds 

Lisa uses a state-of-the-art Transformers based classifier to map natural language into a structured Conversation State. There is a limit on input text length stemming from the quadratic complexity of the attention mechanism in Transformers, as each token (sub word unit) is queried against all other tokens in the input text. A common limit of Transformers-based models is 512 tokens, and to accommodate for this we simply truncate the beginning of the conversation history, as this portion is typically less relevant to the current turn. Recently, linear attention mechanisms have been developed to greatly increase this length limit, but we haven’t found any significant performance gains.

We also include special tokens indicating the speaker, as well as the local timestamp of when each message was sent. This helps Galaxy, by enabling it to infer information from the pacing of messages, as well as the time of day, day of the week and current month. This can also help overcome ambiguities and aid with prioritizing different parts of the input relevant to the current turn.

We generate confidence scores for each label independent of each other to allow an inbound message to have multiple classifications (i.e. a “multi-label” model). This simple setup also allows us to add new labels without touching the model code, we simply modify the data generation, and the new label will show up after the next retraining.

For example, if a prospect asks “I would like a 2 bedroom and do you accept section 8?”, our model will return a score close to 1 for at least two classes – one for asking about “Section 8” (affordable housing) and another for responding with a “2 bedroom” unit type.

Lisa then interprets this state by combining it with external knowledge to generate the natural language response back to the prospect. We refer to the external knowledge as Lisa’s Knowledge Base (KB), and it includes database lookups (e.g. to determine a property’s Section 8 policy) and API calls to external systems (e.g. to Google Calendar for an agent’s availability).

Here is an example of Galaxy in action. Given a message and its conversation history, Galaxy determines a score for each class. Most classes are irrelevant to this message and thus have very low scores. However, Galaxy identified 2 classes of importance here:

1. Updating the unit type to 2 bedrooms

2. The question pertaining to Section 8

When deciding whether the prospect would like a 1 or 2 bedroom apartment, Galaxy paid strong attention to “2 bedroom” in the prospect’s message, but also gave weight to the “1BR” portion in the conversation history. These weights give the class of updating unit type a high score. When judging if there is a Section 8 question, Galaxy gets a strong positive signal from “accept section 8”, but negative signals from the conversations about unit type. This is because prospects don’t tend to mention unit type and Section 8 at the same time. In the end the classifier assigns a positive yet small score to Section 8 class.

Output of the SHAP explainer package for the label SET_UNIT_TYPE_BR2. It shows the importance of each word in the input relating to generating the output score for this label. The colors indicate which parts of the input the model deemed to have positive (red) and negative (blue) contributions. It mostly focuses on the words “2 bedroom” in the last prospect message, but also considers the unit types that Lisa said were available.

Output of the SHAP explainer package for the label SECTION8. It shows the importance of each word in the input relating to generating the output score for this label. The colors indicate which parts of the input the model deemed to have positive (red) and negative (blue) contributions. In this case it mostly focuses on the words “accept section 8?”.

With Lisa’s KB integration we can carry out subsequent actions with our logic layer, such as

• Mark the desired unit type in the database

• Answer the question about the Section 8 policy

• Look up and offer showing slots for the desired unit type

• Cross sell to a different property if the unit type is not available

The logic layer employs a non-ML, template-based approach to generating responses, instead of letting a ML model decide what template to choose or even generate text end-to-end. We chose this methodology because it gives us more control – without having to re-train the model, we can change how Lisa replies to messages or change the conversation flow, just by making adjustments to the logic. Without this, we would need to retrain operators to continuously correct the model’s behavior until enough data is collected to retrain the model, making iterations slow, error prone, and taxing on operators.

Teaching Lisa

To teach Lisa about the leasing process, we need to collect structured training data from the product – one of the greatest challenges underlying all ML products. We carefully designed Lisa’s logic layer to obtain high-quality data without adding much to operators’ workload. Training a classification model usually requires a labeled dataset, one that has annotated classes for each data point.

In our application, this would mean labeling all the desired classes for each inbound message, several hundred labels per message. Instead of asking our operators to create annotations explicitly, we instead infer labels from their behavior. Our operators’ main job is to reply to prospective residents, and to correct our model’s mistakes if needed. 

We implemented a convenient user interface that can provide structured responses for operators to choose from, so our model can learn directly from what operators do on the job. 

One could say that machines learn what we do, not what we say. The user interface needs to account for different categories of classifications, such as question versus intent, and provide operators with easy ways to generate responses by clicking different buttons or navigating the UI with keyboard shortcuts. 

This machine-human interface blurs the boundaries between machine and human responses. Sometimes the machine bypasses operators entirely, and other times operators ignore the suggestions. However, most of the time, the response lies somewhere in the middle; it could be that the machine gives a strong suggestion and operators simply approve it, or that operators slightly modify it to better suit the conversation flow. 

So are prospective tenants talking to a machine or a human? With Lisa, the line is certainly blurry ¯\_(ツ)_/¯

Authors and contributors: Christfried Focke, Shyr-Shea Chang, Tony Froccaro, Miguel Rivera

Lisa: Parsing Inquiries with Seq2Seq Transformers

In our previous blog post we introduced AppFolio AI Leasing Assistant, Lisa, giving a high-level overview of her capabilities and insight into the value she can offer. One of the key technical components which enables Lisa to perform so effectively is the Inquiry Parser.

What is Lisa’s Inquiry Parser?

The leasing conversation flow is often initiated by a prospective resident submitting an inquiry via an Internet Listing Service (ILS), such as Zillow, Apartments.com, or a private homepage for the property. The first component of Lisa to spring into action in response is the inquiry parser. We use it to extract information from inquiries, and process the data collected to start and facilitate a productive conversation in hopes it will lead to a showing, an application, and finally a signed lease.

Once an inquiry is submitted, Lisa receives an e-mail and parses it. All PII (Personal Identifiable Information) is processed and stored securely and not disclosed to anyone not directly involved in the leasing process. At a minimum, a phone number or email address is required to begin a text conversation with the prospective resident. However, with more information such as the prospect’s full name, their desired move-in date, and their unit type preference, Lisa can streamline the conversation as she doesn’t have to ask for it again.

Other than parsing data pertaining to basic information, source attribution is another key component of the inquiry parser. Lisa determines the source of each inquiry, enabling us to generate reports showing which ILS is driving the most business for property managers. 

The Regex Parser has close to 100% precision, but over time its recall will drop as new listing sites come online, or existing sites change their format. We continue to run the RegEx parser first and then augment it with fields from the ML parser. The parsed info is then used to create new, or update existing contacts and threads.

How Does Lisa’s Inquiry Parser work? 

Because there are hundreds of different listing sites, each with different and evolving formats through which they collect their customer’s data, it is a difficult task to parse the wide array of inbound inquiries to Lisa. Prior to the current iteration, our solution was a file with 4,000 lines of RegEx parsing code, that was frequently amended to keep up with formatting changes or addition of new listing sites. This ended up being a significant time sink and chore for our developers. 

Instead, we opted for a more effective solution. In addition to the RegEx, we added a Machine Learning powered parser that generalizes much better by drawing upon data collected from past listing emails and their parsed fields. Lisa now utilizes a Transformers-based, Seq2Seq (sequence-to-sequence) model to map a message derived from an inquiry into a structured string that makes the data trivial to parse. Transformers are a state of the art class of model architectures for Natural Language Processing (NLP) tasks. We leverage pre-trained language models and fine tune them to focus on specific tasks.

As its name suggests, Seq2Seq models transform a sequence into another sequence. A simple example is transforming a German sentence into French. The Transformer generates a target sequence by analyzing both the input and output generated so far, to determine the next token (sub word unit). With the information learned from pre-training on a very large corpus of data, it only needs a fairly modest amount of task specific training data to achieve strong performance.

An illustration of the activations of the attention mechanism that underpins the Transformer architecture. As we are encoding the word "it", part of the attention mechanism is focusing on "The Animal", and baked a part of its representation into the encoding of that word. Source: The Illustrated Transformer.

In our application, we want to extract information from an ILS message. We input the entirety of a message, and have the model output a structured summary sentence of that message. The following is a sample input and output from our model. The input sequence is the ILS message in the top-most text block, the middle text block contains the generated output sequence, and the bottom-most text-block contains the fully parsed output: 

The input sequence consisting of source domain, email subject and body (we remove URLs and HTML tags before passing it into the model) is mapped to a string that resembles natural language and is trivial to parse. We then check whether each value actually exists in the input (e.g. by regex matching phone numbers), and compute confidence scores for each field.

When generating the text in the middle block, the model decides which word to generate based on its relevance to the input text. To explain the model behavior, this relevance can be visualized as a score from each word in the input, and these scores can be added up to determine the final score for the output word (see images below). For example, to generate a part of the phone number, the model almost exclusively looks at the keyword “Phone” and the number that follows. However, when generating the first name, the model actually looks at multiple sentences in the input that mention the first name, even the email address. By looking at these visualizations we can understand how the model works and when its predictions are likely to be correct or incorrect.

Sample output of the SHAP explainer package. It shows the distribution of overall importance when generating part of the phone number (substring “555” in the green circle). The colors indicate which parts of the input the model deemed to have positive (red) and negative (blue) contributions. In this case the model mainly looked at the keyword “Phone” and the phone number itself.

Sample output of the SHAP explainer package. It shows the distribution of overall importance when generating the potential resident’s first name  (substring “Jon” in the green circle). The colors indicate which parts of the input the model deemed to have positive (red) and negative (blue) contributions. In this case the model mainly looked at the keyword(s) “first name”, “Jon” and “My name is Jon…”. 

We chose this model class because the label generation is straight-forward and performance is strong. Lisa simply maps input to target string, and we do not have to annotate exactly where to copy the data from, as would be required of more traditional token classification models. Lisa can read the input and determine the relevant information. There is also no need to post-process the parsed fields to obtain their canonical representation, such as for dates and phone numbers.

One important catch during data generation is that we have to ensure that the value we want to parse is actually present in the source. Otherwise, the model will tend to generate information that is not present. We implemented the same safeguard as a post-processing step, in order to avoid returning occasional “typos.”

In addition to the possibility of typos, another drawback of Seq2Seq models is that there is no obvious way to generate confidence scores. Seq2Seq models output a whole sequence, with the confidence of each predicted word depending on all the previously predicted words in the sentence. This makes it difficult to get a confidence score for the generated sequence or subsequences. Lisa generates confidence scores based on the similarity between the new ILS message and the messages previously used for training the model, as well as the score of the words from which we extract the information.

Lisa’s ML parser has reduced the number of unparsed inquiries to nearly zero and greatly improved the accuracy of data when conducting source attribution. Additionally, the parser has significantly reduced the workload of our operators, who would have had to parse them manually, and our developers who had to maintain the complex parsing code.

The inquiry parser is just one of many exciting components that make up Lisa. Stay tuned for the next post, as we deep dive into the main driver of our conversational system that will leave you questioning whether or not you are actually speaking to an AI.

Authors and contributors: Christfried Focke, Shyr-Shea Chang, Tony Froccaro, Miguel Rivera

AppFolio’s AI Leasing Assistant, Lisa

Let’s delve into the past and think about the last time you tried to rent an apartment. Hopefully, this doesn’t trigger any painful memories…

1. First, you went to one of the familiar listing sites — Apartments.com, Zillow, or maybe even Craigslist if you’re the type to live dangerously.

2. You sent a message or made a phone call to the places that piqued your interest.

3. You heard back from some of the inquiries, but from others - silence. You might have even ended up on the phone with a grumpy landlord who acted as if you were wasting their time, or with an intern who couldn’t provide you with an ounce of concrete information.

4. Once you made a connection, you scheduled a showing to take a look at the place. Most likely, this was coordinated over text, email, a phone call, or social media.

5. You eventually found a place, but you were still left with a lingering sense of doubt. You may have missed out on another fantastic place to rent just because of a communication breakdown.

On paper, the process of finding housing sounds simple and relatively straightforward. However, in practice, it’s often a convoluted process –  one that includes many hardships and unpredictability. In fact, only 5% of all rental inquiries end with the filing of an application. Put shortly, the leasing process often results in a large amount of wasted effort, both for prospective residents and property managers. 

Traditionally, the point of contact for prospective renters has been a leasing agent who engages in time-consuming and repetitive conversations with potential renters. Because so many of these conversations follow a predictable script, AppFolio identified a unique opportunity to streamline the process through automation. 

AppFolio AI Leasing Assistant, Lisa, is AppFolio’s response to this problem. Lisa outsources the repetitive and time-consuming parts of leasing conversations while retaining human operators to cover the long tail and provide training data for future automation. Below is a diagram showcasing the processes Lisa is capable of carrying out, and a mock scenario of Lisa at work.

Typical flow of a leasing conversation, Blue boxes indicate steps that are automated by Lisa.

Initial Onboarding of a Prospective Resident

We’ll start with the customer. George is looking for a new place and browses familiar listing sites. Eventually, he sends out a few inquiries. One of these inquiries makes it to Lisa, who automatically parses out George’s profile information, and stores George’s interest in the property’s sales database. Lisa will also use this information to initiate a text conversation with George in less than one or two minutes.

Answer Questions, Prompt for Sales Information

George has come prepared and wants to ask a series of questions most prospective renters ask when in search of a new home:

“Are pets allowed in the apartment? What is the estimated price of utilities per month? What is the floor plan of the available units?”

Lisa can detect and answer these common questions automatically by drawing on a standardized set of policies the property sets. Lisa will also nudge the conversation forward. She’ll request additional contact information and suggest times when George might come by the apartment for a tour.

LISA CONVERTS THE NATURAL LANGUAGE INTO A STRUCTURED RESPONSE THAT ALLOWS THE LOGIC LAYER TO COMPUTE THE RESPONSE.

Find a Showing Time and Handle Scheduling Conflicts

George would like a showing, but he works long hours and can only attend a showing during the evening or on weekends. This could be a problem, not only for George but also for the property managers, since agents have limited availability on the weekends, and agents are sometimes booked weeks in advance. George tells Lisa his availability and Lisa immediately cross-checks both parties’ schedules to find a time that works.

LISA PARSES THE SHOWING TIME PREFERENCE AND CAN RESPOND TO COUNTEROFFERS.

Lisa’s value proposition should be clear: Lisa can maintain any number of parallel conversations with prospective renters, provide excellent customer service, and free up time for property managers to focus less on the minutiae and more on the big picture. 

Overview: How Does Lisa Work? 

AppFolio retains a staff of operators, who get a chance to review conversations as they unfold, rapidly re-label messages on the fly, and use tools to handle edge cases or language that our current models fail to understand. 

Lisa uses a collection of concepts and models to achieve these outcomes including:

• A parser for inquiries from Internet Listing Services (ILS)

• A dialog system that combines

  • Natural Language Understanding (NLU) 

  • Dialogue State Tracking 

  • Policy (a logic layer that combines conversation state and external knowledge to decide the next step) 

  • Template-based Natural Language Generation (NLG)

• A recommendation system for cross-sells

• A forecasting model to determine hiring goals and set staffing levels for each shift

Stay tuned as we will discuss each of these components in greater detail as part of a series of blog posts surrounding the nuances and technologies!

Lisa integrates with several external systems via APIs and Natural language.

Authors and contributors: Christfried Focke, Shyr-Shea Chang, Tony Froccaro, Ian Murray