Text Analytics Toolbox: Complete reference

Complete specification of worksheet controls, model registry, rule syntax, output schemas, diagnostic thresholds, and troubleshooting guidelines.

← Text Analytics Toolbox overview · Getting started · Modeling guide · Methods guide · Examples

Worksheet control catalog (Start!A5:B10)

The Start worksheet hosts runtime parameters governing task routing, preprocessing, and vectorization:

Cell Control Name Type Allowed Values Default Algorithmic Impact
B6 Active task Dropdown Any registered Model ID customer_reviews_sentiment Routes execution to the selected NLP engine, filtering Corpus and Stopwords & Rules tables.
B7 Num topics (K) Integer \(2 \le K \le 10\) 3 Sets number of latent Dirichlet topics for the LDA model. Clamped internally to \([2, 10]\) and \(n_{\text{docs}}\).
B8 Top terms count Integer \(3 \le k \le 20\) 8 Determines number of top TF-IDF keywords and vocabulary features published to Vocabulary & Topics.
B9 Min doc freq Integer \(\ge 1\) 1 Filters terms appearing in fewer than this number of documents. Higher values eliminate rare singletons.
B10 Case fold Dropdown Lowercase, Preserve Lowercase Controls case normalization during regex tokenization and stopword matching.

Model catalog (Start!D5:H9)

The model library registers built-in task configurations:

Model ID (Col D) Label (Col E) Family (Col F) Algorithm (Col G) Description (Col H)
customer_reviews_sentiment Product Reviews — Sentiment Sentiment Lexicon Extract polarity and sentiment scores from product customer reviews with negation handling.
support_tickets_topics Support Tickets — Topic Model Topics LDA Unsupervised topic discovery across technical support tickets using variational Bayes.
keyword_tfidf_extraction Document Corpus — TF-IDF Keywords Tfidf Extract highest-ranking discriminative terms across documents via sublinear TF-IDF.
document_similarity_search Content Library — Search Similarity Cosine Query search matching document cosine similarity scores with match tiering.

Stopwords & Rules reference (Stopwords & Rules!A5:D50)

The preprocessing rules table configures NLP tokenization directives:

Column Header Type Description and Allowed Values
A Model Text Model scope for rule. Set to ALL to apply across every model, or enter a specific Model ID to restrict scope.
B RuleType Text Directive type: Stopword, MinLength, PositiveWords, NegativeWords, NgramMax, SearchQuery.
C Value Mixed Rule argument. Comma-separated list for words, positive integer for lengths and n-grams, text for queries.
D Description Text Human-readable explanation of rule purpose and operational intent.

Rule syntax rules

  • Stopword: Comma-separated words ("ticket, issue, error"). Whitespace is automatically stripped. Tokens are case-matched according to Start!B10.
  • MinLength: Single positive integer (\(\ge 1\)). Default is 3. Tokens shorter than this length are filtered by regex (?u)\b\w{min_length,}\b.
  • PositiveWords / NegativeWords: Comma-separated domain sentiment terms. Merged with built-in baseline affective lexicons.
  • NgramMax: Integer \(1\), \(2\), or \(3\). Controls the maximum n-gram order for TF-IDF extraction.
  • SearchQuery: Free-form text query string converted into TF-IDF vector for cosine similarity ranking.

Results output schemas

Summary block schema (Results!A5:B17)

The 12-row summary table provides high-level corpus and performance metrics:

Row Metric Header (Col A) Value Type (Col B) Description
6 Active task Text Descriptive label of the active task from the model catalog.
7 Algorithm Text Active algorithm (Lexicon, LDA, Tfidf, Cosine).
8 Corpus status Text Ingestion status (OPTIMAL).
9 Document count Integer Total documents ingested for the active model (\(n\)).
10 Total tokens Integer Total clean tokens across all ingested documents.
11 Unique vocabulary Integer Distinct vocabulary terms after length and stopword filtering (\(|\mathcal{V}|\)).
12 Mean doc length Number (2 decimals) Average tokens per document: \(N_{\text{total}} / n\).
13 Metric 8 (Dynamic) Mixed Task-specific metric (see table below).
14 Metric 9 (Dynamic) Mixed Task-specific metric (see table below).
15 Metric 10 (Dynamic) Mixed Task-specific metric (see table below).
16 Matrix sparsity (%) Percentage (2 dec) Zero-entry percentage of document-term matrix.
17 Runtime (s) Number (4 decimals) Monotonic Python execution time in seconds.

Dynamic rows (Rows 13–15) by active task

Active Task Row 13 (Metric 8) Row 14 (Metric 9) Row 15 (Metric 10)
customer_reviews_sentiment Dominant polarity (Positive / Negative) Positive documents (Count) Negative documents (Count)
support_tickets_topics Num topics (K) (Integer) Dominant topic (Topic X) Avg topic confidence (Percentage, e.g. 92.5%)
keyword_tfidf_extraction Max n-gram order (Integer) Top corpus keyword (Text) Mean top TF-IDF (Score, e.g. 0.2142)
document_similarity_search Search query (Truncated text) High matches (≥0.4) (Count) Top match doc (Doc X.X (Score))

Document scoring schema (Results!D5:I45)

Columns D, E, and F remain constant across all tasks: - D: DocID (Identifier passed from Corpus sheet) - E: Category (Categorical tag passed from Corpus sheet) - F: TokenCount (Clean token count in document)

Columns G, H, and I reflect task-specific scoring:

Column Sentiment Analysis Topic Modeling (LDA) TF-IDF Keyword Extraction Cosine Similarity Search
G SentimentScore (\(-1.0\) to \(+1.0\)) TopicWeight (\(0.0\) to \(1.0\)) TopTfidfScore (Float) SimilarityScore (\(0.0\) to \(1.0\))
H Polarity (Pos / Neg / Neut) DominantTopic (Topic X) TopKeyword (String) MatchTier (High / Mod / Low)
I DominantTopic (Topic X) TopKeywords (String list) KeyNgrams (String list) TopMatchedTerms (String list)

Vocabulary & Topics schema (Vocabulary & Topics!A5:F40)

The vocabulary table provides granular feature and topic weights:

Column Header Type Description
A ItemType Text Feature category: Keyword, LexiconPos, LexiconNeg, Topic, CorpusKeyword, DocKey, QueryTerm, CorpusTerm.
B Identifier Text Sequential label: Term 1, Pos 1, Neg 1, Topic 1, Keyword 1, Doc 1, Query 1.
C Term Text Word, n-gram phrase, or comma-separated top topic terms.
D Score Number Normalized mathematical weight, TF-IDF score, or topic component sum.
E Weight Number Scaled percentage weight (\(\text{Score} \times 100\)) or topic prevalence percentage.
F Rank Integer Numerical rank within item category (\(1, 2, 3\dots\)).

Diagnostics reference table

The automated 7-check Diagnostics Suite evaluates corpus health on Diagnostics!A5:C25:

Check Name Severity Passing Condition Warning Condition Remediation Guidance
1. Corpus ingestion PASS / WARN \(n_{\text{docs}} \ge 4\) \(n_{\text{docs}} < 4\) Add more document rows for the active model on the Corpus sheet.
2. Vocabulary filtering PASS / WARN \(|\mathcal{V}| \ge 10\) \(|\mathcal{V}| < 10\) Vocabulary is overly constricted. Lower MinLength or decrease Min doc freq.
3. Matrix sparsity PASS / WARN \(\text{Sparsity} < 99.5\%\) \(\text{Sparsity} \ge 99.5\%\) Documents share virtually no vocabulary. Check for language mismatches or increase corpus size.
4. Lexical diversity PASS / WARN \(0.10 \le \text{TTR} \le 0.95\) \(\text{TTR} < 0.10\) or \(\text{TTR} > 0.95\) TTR < 0.10 indicates repetitive text spam; TTR > 0.95 indicates isolated singletons without common terms.
5. Sentiment & query coverage PASS / WARN Affective hits \(> 0\) (Sentiment), Count non-zeros \(> 0\) (LDA), Peak TF-IDF \(> 0.1\) (Keywords), Query match \(> 0\) (Search) Zero matches or coverage Add domain affective words to Stopwords & Rules, or revise the search query to include corpus terms.
6. Topic model balance PASS / WARN Both pos & neg docs present (Sentiment), Min topic prev \(\ge 5.0\%\) (LDA), Mean top TF-IDF \(> 0.15\) (Keywords), \(\ge 1\) high/mod match (Search) Skewed distribution, degenerate topics, or zero query matches For LDA, decrease \(K\) if topics collapse. For Sentiment, check that negative feedback exists. For Search, refine query.
7. Rule configuration PASS / WARN Rules parsed successfully Parsing exception in rules Verify syntax on Stopwords & Rules; ensure numbers are valid integers.

Error handling & troubleshooting guide

1. ValueError: Unknown active task '<task_id>'

  • Cause: Cell Start!B6 contains a model name not found in column D of the model catalog (Start!D5:H9).
  • Fix: Select a valid model ID from the cell dropdown list, or add the new model key to the catalog table.

2. ValueError: No document rows found for model '<task_id>' on Corpus sheet

  • Cause: The active task has no matching rows in Corpus!A6:A200.
  • Fix: Verify the spelling of the Model column in the Corpus sheet. Column A must exactly match Start!B6.

3. High sparsity warning (Sparsity >= 99.5%)

  • Cause: Document texts are very short and use completely disjoint vocabularies.
  • Fix: Remove non-essential stopwords, lower min_df to 1, or ensure documents belong to related semantic topics.

4. Degenerate LDA topic distributions (all documents assigned to one topic)

  • Cause: The topic count \(K\) is set too high for a small corpus, or min_df filtered out all shared words.
  • Fix: Reduce Num topics (K) in Start!B7 to 2 or 3, and confirm domain-specific boilerplate words are added to stopwords.

5. Cosine similarity produces all zeros (Score = 0.0000)

  • Cause: None of the terms in the search query exist in the corpus vocabulary after stopword and length filtering.
  • Fix: Inspect the active query in Stopwords & Rules!C11. Ensure query words are spelled correctly and appear in the documents. Check Check 5 on Diagnostics to confirm matched query term counts.