Text Analytics Toolbox: Methods guide

Mathematical formulations of negation-aware lexicon sentiment, sublinear TF-IDF, variational Latent Dirichlet Allocation, cosine similarity, and diagnostics methodology.

← Text Analytics Toolbox overview · Getting started · Modeling guide · Examples · Reference

Mathematical formulation

The Text Analytics Toolbox implements quantitative natural language processing algorithms covering lexicon-based sentiment analysis, frequency-inverse document frequency weighting, unsupervised probabilistic topic discovery, and vector space similarity search.

1. Lexicon sentiment analysis with negation inversion

The sentiment analysis engine calculates a normalized continuous polarity score \(s \in [-1.0, 1.0]\) for each document \(d\):

\[s(d) = \frac{N_{\text{pos}}(d) - N_{\text{neg}}(d)}{\max(N_{\text{tokens}}(d), 1)}\]

where: - \(N_{\text{tokens}}(d)\) is the total count of clean tokens in document \(d\) satisfying length and stopword thresholds. - \(N_{\text{pos}}(d)\) is the effective count of positive affective terms. - \(N_{\text{neg}}(d)\) is the effective count of negative affective terms.

Negation lookback window and clause isolation

In natural language, negation markers radically invert semantic polarity (e.g., “not good”, “never recommended”, “without exceeding expectations”). The engine processes tokens sequentially within clause boundaries:

  1. Clause Delimiters: Sentence punctuation (., !, ?, ;) resets the negation state: \[\text{State}_{\text{negated}} \leftarrow \text{False}\]
  2. Negator Detection: If a token matches the negation set \(\mathcal{M}_{\text{neg}} = \{\text{not}, \text{never}, \text{no}, \text{neither}, \text{nor}, \text{without}\}\) or ends with the contraction suffix n't, the negator flag is activated: \[\text{State}_{\text{negated}} \leftarrow \text{True}\]
  3. Polarity Inversion:
    • If an affective word \(w \in \mathcal{V}_{\text{pos}}\) is encountered while \(\text{State}_{\text{negated}} = \text{True}\): \[N_{\text{neg}} \leftarrow N_{\text{neg}} + 1, \quad \text{State}_{\text{negated}} \leftarrow \text{False}\]
    • If an affective word \(w \in \mathcal{V}_{\text{neg}}\) is encountered while \(\text{State}_{\text{negated}} = \text{True}\): \[N_{\text{pos}} \leftarrow N_{\text{pos}} + 1, \quad \text{State}_{\text{negated}} \leftarrow \text{False}\]
    • Unnegated affective words increment their natural counters (\(N_{\text{pos}}\) or \(N_{\text{neg}}\)).

Categorical polarity thresholds

Documents are mapped into discrete bins using an indifference deadband around zero:

\[\text{Polarity}(d) = \begin{cases} \text{Positive}, & s(d) > +0.05 \\ \text{Negative}, & s(d) < -0.05 \\ \text{Neutral}, & -0.05 \le s(d) \le +0.05 \end{cases}\]

2. Sublinear TF-IDF feature extraction

Term Frequency-Inverse Document Frequency (TF-IDF) balances term local importance within a document against corpus-wide rarity.

Sublinear term frequency scaling

Raw term frequency \(\text{tf}(t, d)\) can bias document representations toward repetitive words. The toolbox applies logarithmic sublinear term frequency scaling (sublinear_tf=True):

\[\text{tf}_{\text{sub}}(t, d) = \begin{cases} 1 + \ln(\text{tf}(t, d)), & \text{tf}(t, d) > 0 \\ 0, & \text{otherwise} \end{cases}\]

This logarithmic compression ensures that twenty occurrences of a term do not exert twenty times the influence of a single occurrence.

Smooth inverse document frequency

Inverse document frequency measures discriminatory power across the collection of \(n\) documents:

\[\text{idf}(t) = \ln\left(\frac{1 + n}{1 + \text{df}(t)}\right) + 1\]

where \(\text{df}(t) = |\{d \in \mathcal{D} : t \in d\}|\) is the document frequency (the number of documents containing term \(t\)). The addition of \(1\) inside the logarithm prevents division by zero, and the outer \(+1\) ensures that terms appearing in all documents retain non-zero weight rather than being discarded entirely.

N-gram feature union and \(L_2\) normalization

When NgramMax = 2, the feature space \(\mathcal{V}\) expands to include both unigrams (\(w_i\)) and bigrams (\(w_i w_{i+1}\)):

\[\mathcal{V} = \mathcal{V}_{\text{unigram}} \cup \mathcal{V}_{\text{bigram}}\]

The un-normalized TF-IDF weight vector for document \(d\) is:

\[\mathbf{v}_d(t) = \text{tf}_{\text{sub}}(t, d) \times \text{idf}(t)\]

To eliminate document length bias, each row vector is projected onto the unit hypersphere via Euclidean \(L_2\) normalization:

\[\mathbf{x}_d = \frac{\mathbf{v}_d}{\|\mathbf{v}_d\|_2} = \frac{\mathbf{v}_d}{\sqrt{\sum_{t \in \mathcal{V}} \mathbf{v}_d(t)^2}}\]

3. Latent Dirichlet Allocation (LDA) topic modeling

Latent Dirichlet Allocation is a generative probabilistic Bayesian model that decomposes a collection of discrete documents into \(K\) latent semantic topics.

Generative probabilistic model

For each document \(d \in \{1, \dots, n\}\): 1. Draw document topic proportions \(\boldsymbol{\theta}_d \sim \text{Dirichlet}(\boldsymbol{\alpha})\), where \(\boldsymbol{\alpha}\) is a \(K\)-dimensional symmetric prior parameter. 2. For each word position \(i \in \{1, \dots, N_d\}\): - Sample a topic assignment \(z_{d, i} \sim \text{Multinomial}(\boldsymbol{\theta}_d)\). - Sample an observed word \(w_{d, i} \sim \text{Multinomial}(\boldsymbol{\beta}_{z_{d, i}})\), where \(\boldsymbol{\beta}_k\) is the term distribution over vocabulary \(\mathcal{V}\) for topic \(k\), drawn from \(\text{Dirichlet}(\boldsymbol{\eta})\).

The marginal probability of document \(d\) is:

\[p(d \mid \boldsymbol{\alpha}, \boldsymbol{\beta}) = \int p(\boldsymbol{\theta}_d \mid \boldsymbol{\alpha}) \left( \prod_{i=1}^{N_d} \sum_{z_{d, i}=1}^K p(z_{d, i} \mid \boldsymbol{\theta}_d) \, p(w_{d, i} \mid z_{d, i}, \boldsymbol{\beta}) \right) d\boldsymbol{\theta}_d\]

Variational Bayes inference on discrete counts

Because the true posterior distribution is intractable, the toolbox solves LDA using online Variational Expectation-Maximization (variational Bayes) in scikit-learn.

[!IMPORTANT] Why discrete counts are required instead of TF-IDF: LDA is fundamentally a generative multinomial count model. Its likelihood function is defined over discrete word occurrence counts \(C(w, d) \in \mathbb{N}_0\). Passing continuous, \(L_2\)-normalized TF-IDF values violates the multinomial distribution assumptions and causes variational inference to fail or produce distorted topic distributions. Therefore, the toolbox extracts discrete term counts via CountVectorizer for LDA while using TfidfVectorizer for keyword extraction and cosine similarity.

Document topic mixture and topic dominance

The fitted model outputs: - \(\boldsymbol{\theta}_{d, k}\): The probability that document \(d\) belongs to latent topic \(k\), satisfying \(\sum_{k=1}^K \boldsymbol{\theta}_{d, k} = 1.0\). - Dominant topic: \(k^* = \arg\max_k \boldsymbol{\theta}_{d, k}\). - Topic confidence: \(\max_k \boldsymbol{\theta}_{d, k}\).

Corpus-wide topic prevalence is computed by averaging probabilities across all documents:

\[\bar{\theta}_k = \frac{1}{n} \sum_{d=1}^n \boldsymbol{\theta}_{d, k} \times 100\%\]

4. Cosine similarity search and match tier boundaries

Given an ad-hoc multi-word search query \(q\), the query text is transformed into the document feature space using the fitted corpus TF-IDF vectorizer:

\[\mathbf{q} = \text{vec.transform}([q]) \in \mathbb{R}^{|\mathcal{V}|}\]

Because both \(\mathbf{q}\) and document vectors \(\mathbf{d}_i\) are \(L_2\)-normalized (\(\|\mathbf{q}\|_2 = 1\) and \(\|\mathbf{d}_i\|_2 = 1\)), the cosine similarity reduces to the Euclidean dot product:

\[\cos(\mathbf{q}, \mathbf{d}_i) = \frac{\mathbf{q} \cdot \mathbf{d}_i}{\|\mathbf{q}\|_2 \|\mathbf{d}_i\|_2} = \sum_{t \in \mathcal{V}} q(t) \, d_i(t) \quad \in [0.0, 1.0]\]

Match tier classification

Documents are categorized into three practical decision tiers based on cosine score:

\[\text{MatchTier}(\mathbf{d}_i) = \begin{cases} \text{High } (\ge 0.4), & \cos(\mathbf{q}, \mathbf{d}_i) \ge 0.40 \\ \text{Moderate } (0.2\text{--}0.4), & 0.20 \le \cos(\mathbf{q}, \mathbf{d}_i) < 0.40 \\ \text{Low } (< 0.2), & \cos(\mathbf{q}, \mathbf{d}_i) < 0.20 \end{cases}\]

Diagnostics Suite methodology

The 7-check Diagnostics Suite evaluates corpus quality and algorithmic health across seven statistical and structural tests:

flowchart TD
    subgraph Suite["7-Check Diagnostics Suite"]
        C1["1. Corpus Ingestion<br/>n_docs >= 4"]
        C2["2. Vocabulary Filtering<br/>vocab_size >= 10"]
        C3["3. Matrix Sparsity<br/>sparsity < 99.5%"]
        C4["4. Lexical Diversity<br/>0.10 <= TTR <= 0.95"]
        C5["5. Sentiment/Query Coverage<br/>Hits > 0"]
        C6["6. Topic/Sentiment Balance<br/>Pos & Neg > 0 / Prev >= 5%"]
        C7["7. Rule Configuration<br/>Valid rules count"]
    end

Check 1: Corpus ingestion

  • Metric: Document count \(n = |\mathcal{D}|\).
  • Condition: \(\text{PASS}\) if \(n \ge 4\); otherwise \(\text{WARN}\).
  • Rationale: Topic modeling and TF-IDF require sufficient document collections to establish baseline frequency distributions.

Check 2: Vocabulary filtering

  • Metric: Post-filtering active vocabulary size \(|\mathcal{V}|\).
  • Condition: \(\text{PASS}\) if \(|\mathcal{V}| \ge 10\); otherwise \(\text{WARN}\).
  • Rationale: Overly aggressive stopwords or high min_df thresholds can collapse the feature space, leaving models without discriminatory tokens.

Check 3: Matrix sparsity

  • Metric: Percentage of zero entries in the document-term matrix: \[\text{Sparsity} = \left(1 - \frac{\text{nnz}(\mathbf{X})}{n \times |\mathcal{V}|}\right) \times 100\%\]
  • Condition: \(\text{PASS}\) if \(\text{Sparsity} < 99.5\%\); otherwise \(\text{WARN}\).
  • Rationale: Sparsity exceeding 99.5% indicates that documents share virtually no common vocabulary, impeding topic convergence and similarity search.

Check 4: Lexical diversity (Type-Token Ratio)

  • Metric: Type-Token Ratio (\(\text{TTR}\)): \[\text{TTR} = \frac{|\text{Unique Tokens}|}{N_{\text{total}}}\]
  • Condition: \(\text{PASS}\) if \(0.10 \le \text{TTR} \le 0.95\); otherwise \(\text{WARN}\).
  • Rationale: A \(\text{TTR} < 0.10\) indicates degenerate repetitive text (e.g., automated error spam), while \(\text{TTR} > 0.95\) indicates that every word appears only once (zero lexical overlap).

Check 5: Sentiment & query coverage

  • Metric: Task-specific feature coverage:
    • Sentiment: Total lexicon hits \(N_{\text{pos}} + N_{\text{neg}} > 0\).
    • LDA Topics: Discrete count matrix non-zero entries \(\text{nnz}(\mathbf{C}) > 0\).
    • TF-IDF: Peak document top TF-IDF score \(> 0.10\).
    • Similarity Search: Query terms matching active vocabulary \(> 0\).
  • Condition: \(\text{PASS}\) if threshold met; otherwise \(\text{WARN}\).

Check 6: Topic model & sentiment balance

  • Metric: Task-specific distributional balance:
    • Sentiment: Both Positive count \(> 0\) and Negative count \(> 0\).
    • LDA Topics: Minimum topic prevalence \(\min_k \bar{\theta}_k \ge 5.0\%\).
    • TF-IDF: Mean top document TF-IDF score \(> 0.15\).
    • Similarity Search: At least one document in High (\(\ge 0.4\)) or Moderate (\(0.2\text{--}0.4\)) tier.
  • Condition: \(\text{PASS}\) if balanced; otherwise \(\text{WARN}\).

Check 7: Rule configuration

  • Metric: Count of successfully parsed rules from Stopwords & Rules.
  • Condition: \(\text{PASS}\) if rules parsed without fatal configuration errors.

MATLAB Text Analytics Toolbox comparison & mapping

For engineers and researchers transitioning from MATLAB’s Text Analytics Toolbox, the table below maps MATLAB functions to Boardflare’s implementation:

MATLAB Text Analytics Function MATLAB Purpose Boardflare Python for Excel Implementation
tokenizedDocument Tokenizes character arrays or strings Regex word-boundary tokenizer (?u)\b\w{min_length,}\b
bagOfWords Creates discrete word count matrix sklearn.feature_extraction.text.CountVectorizer
removeStopWords Filters common function words Union of scikit-learn built-in English stopwords + custom rules
tfidf Calculates TF-IDF sparse matrix sklearn.feature_extraction.text.TfidfVectorizer(sublinear_tf=True)
fitlda Fits Latent Dirichlet Allocation model sklearn.decomposition.LatentDirichletAllocation(n_components=K)
vaderSentimentScores Rule-based sentiment scoring Lexicon polarity scoring with sentence-clause negation inversion
docsearch Cosine similarity document indexing sklearn.metrics.pairwise.cosine_similarity with match tiering
wordcloud Visualizes term frequency cloud Horizontal bar chart of top TF-IDF weights and topic distributions