Recently Published
Publish Document
3. Interesting Findings
Corpus Differences: Blog posts tend to be longer and more conversational, news articles follow formal journalistic structure, and Twitter posts are much shorter with unique slang and abbreviations.
Zipf's Law: A very small number of unique words (like stop words "the", "and", "to") make up a massive percentage of total word occurrences across all three datasets.
Memory Constraints: Due to the large volume of text data, efficient sampling and base R vector operations are essential to prevent system memory exhaustion during processing.
4. Plan for Prediction Algorithm and Shiny App
The Prediction Algorithm: We plan to build an N-gram back-off model (using trigrams and 4-grams) with frequency tables. If a longer word combination isn't found, the model will back off to smaller n-grams to predict the next word accurately.
The Shiny App: We will deploy an interactive web application where users can type any phrase into a text box and instantly view the top predicted next words in real time. We welcome feedback on balancing model accuracy with fast response speeds!