{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Homework 4 -- Name(s) here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Due Wednesday, November 30 by midnight**. You may submit this assignment in groups of 2. Be sure to put your names above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this homework, we approach the problem of identifying a small subset of a dataset using unsupervised and supervised methods. The dataset we'll use is a set of newsgroup postings from the early days of the internet.\n", "\n", "In a real world application, we often don’t have labels, and clustering and outlier detection are usually applied in settings that don’t have labels. For the clustering part of this homework, you should work without the ground truth labels as much as possible. Often inspecting and visualizing the data is the only way to understand the result of clustering.\n", "\n", "However, since we do have ground-truth we could do a post-hoc analysis and determine how well we actually did. You might explore that after you've done the clustering." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rec.sport.hockey 600\n", "soc.religion.christian 599\n", "rec.motorcycles 598\n", "rec.sport.baseball 597\n", "sci.crypt 595\n", "rec.autos 594\n", "sci.med 594\n", "sci.space 593\n", "comp.windows.x 593\n", "sci.electronics 591\n", "comp.os.ms-windows.misc 591\n", "comp.sys.ibm.pc.hardware 590\n", "misc.forsale 585\n", "comp.graphics 584\n", "comp.sys.mac.hardware 578\n", "talk.politics.mideast 564\n", "talk.politics.guns 546\n", "alt.atheism 480\n", "talk.politics.misc 465\n", "talk.religion.misc 377\n", "Name: Category Name, dtype: int64\n", "I was wondering if anyone out there could enlighten me on this car I saw\n", "the other day. It was a 2-door sports car, looked to be from the late 60s/\n", "early 70s. It was called a Bricklin. The doors were really small. In addition,\n", "the front bumper was separate from the rest of the body. This is \n", "all I know. If anyone can tellme a model name, engine specs, years\n", "of production, where this car is made, history, or whatever info you\n", "have on this funky looking car, please e-mail.\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TextCategory LabelCategory Name
0I was wondering if anyone out there could enli...7rec.autos
1A fair number of brave souls who upgraded thei...4comp.sys.mac.hardware
2well folks, my mac plus finally gave up the gh...4comp.sys.mac.hardware
3\\nDo you have Weitek's address/phone number? ...1comp.graphics
4From article <C5owCB.n3p@world.std.com>, by to...14sci.space
............
11309DN> From: nyeda@cnsvax.uwec.edu (David Nye)\\nD...13sci.med
11310I have a (very old) Mac 512k and a Mac Plus, b...4comp.sys.mac.hardware
11311I just installed a DX2-66 CPU in a clone mothe...3comp.sys.ibm.pc.hardware
11312\\nWouldn't this require a hyper-sphere. In 3-...1comp.graphics
11313Stolen from Pasadena between 4:30 and 6:30 pm ...8rec.motorcycles
\n", "

11314 rows × 3 columns

\n", "
" ], "text/plain": [ " Text Category Label \\\n", "0 I was wondering if anyone out there could enli... 7 \n", "1 A fair number of brave souls who upgraded thei... 4 \n", "2 well folks, my mac plus finally gave up the gh... 4 \n", "3 \\nDo you have Weitek's address/phone number? ... 1 \n", "4 From article , by to... 14 \n", "... ... ... \n", "11309 DN> From: nyeda@cnsvax.uwec.edu (David Nye)\\nD... 13 \n", "11310 I have a (very old) Mac 512k and a Mac Plus, b... 4 \n", "11311 I just installed a DX2-66 CPU in a clone mothe... 3 \n", "11312 \\nWouldn't this require a hyper-sphere. In 3-... 1 \n", "11313 Stolen from Pasadena between 4:30 and 6:30 pm ... 8 \n", "\n", " Category Name \n", "0 rec.autos \n", "1 comp.sys.mac.hardware \n", "2 comp.sys.mac.hardware \n", "3 comp.graphics \n", "4 sci.space \n", "... ... \n", "11309 sci.med \n", "11310 comp.sys.mac.hardware \n", "11311 comp.sys.ibm.pc.hardware \n", "11312 comp.graphics \n", "11313 rec.motorcycles \n", "\n", "[11314 rows x 3 columns]" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "import numpy as np\n", "from sklearn.datasets import fetch_20newsgroups\n", "newsgroups = fetch_20newsgroups(remove=('headers', 'footers', 'quotes'))\n", "\n", "all_df = pd.DataFrame({'Text': newsgroups.data, 'Category Label': newsgroups.target})\n", "all_df['Category Name'] = all_df['Category Label'].map(lambda idx: newsgroups.target_names[idx])\n", "print(all_df['Category Name'].value_counts())\n", "print(all_df['Text'][0])\n", "all_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 0: Split the Data (2 points)\n", "\n", "Choose any 3-6 categories. Create a new dataframe containing only the rows in those categories." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 1: Vectorization (6 points)\n", "\n", "## 1.1 Plain Tfidf\n", "\n", "Vectorize the data using TfIdf. Do not change the defaults. What are the top 10 most 'important' (highest tfidf score) words?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.2 Better Tfidf\n", "\n", "Now vectorize again but this time choose reasonable values for some of the parameters, such as stop_words. What are the top 10 most 'important' terms now?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 2: Dimensionality Reduction and Visualization (8 points)\n", "\n", "## 2.1 PCA\n", "\n", "Use PCA to reduce your \"Better Tfidf\" data to 2 dimensions and plot the data. Color each point by its category. Make sure the plot has a useful legend.\n", "\n", "Do you see anything notable?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.2 T-SNE\n", "\n", "Now do the same with T-SNE. See if tuning the perplexity parameter helps obtaining a better visualization.\n", "\n", "Do you see anything notable?\n", "\n", "Note: You may want to re-run Tfidf and set the max_features parameter to some reasonable value (e.g., 100). Otherwise TSNE may take a very long time." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Task 3: Clustering (12 points)\n", "\n", "Now we will cluster the Tfidf-vectorized data. (Do NOT cluster on the PCA- or TSNE-reduced data.) For each algorithm, try to manually tune the parameters for a reasonable outcome and document how you tuned the parameters. In particular pay attention to the sizes of the clusters created.\n", "\n", "Manually inspect the outcomes as well as you can and identify if any of the resulting clusters are meaningful (as far as you can tell).\n", "\n", "### 3.1 KMeans\n", "\n", "Use KMeans to cluster the data. Use the PCA or TSNE projections (your choice) to make another scatterplot, but this time color the points by their cluster label from KMeans.\n", "\n", "On the same plot, plot the cluster centers that KMeans found. This will mean you'll need to use your fit PCA or TSNE model to transform into 2 dimensions. Plot these with a different marker shape.\n", "\n", "For each cluster center, find the 5 closest posts and print them. Do you notice anything useful? Hint: `KMeans` has a transform method that might be useful. Hint 2: `np.argsort` is your friend." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2 Agglomerative Clustering\n", "\n", "Use Agglomerative Clustering (with ward linkage) to cluster the data. Create a dendrogram for agglomerative clustering (the truncate_mode='level' might be useful).\n", "\n", "Manually inspect the outcomes as well as you can and identify if any of the resulting clusters are meaningful (as far as you can tell)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.3 DBSCAN\n", "\n", "Use DBSCAN to cluster the data. Make a scatterplot of the results using the PCA or TSNE transformations. You don't need to worry about plotting the cluster centers in this case.\n", "\n", "Manually inspect the outcomes as well as you can and identify if any of the resulting clusters are meaningful (as far as you can tell)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 4: Classification (12 points)\n", "\n", "Now it's time to see if we can build a classifier.\n", "\n", "## 4.1 Baseline\n", "\n", "Pick a linear model and create a simple baseline classifier.\n", "\n", "What words/terms does the model think are most important? Try to interpret your model." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.2 Linear Model\n", "\n", "Try to improve your baseline by trying several (2-3) different linear classifiers, making sure to carefully tune hyperparameters and use cross-validation.\n", "\n", "What words/terms does your best model think are most important? Try to interpret your model.\n", "\n", "Optional: You are welcome to add extra features you think might be useful. That might be the count of a particular word in each post (e.g., a popular misspelling of a word), whether or not a particular email address is mentioned, etc. There is not a ton of room for customization here, but there are some things to try." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4.3 Non-Linear Model\n", "\n", "Now do the same as above but use non-linear models. Try out 2-3 of them." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.5" }, "vscode": { "interpreter": { "hash": "81794d4967e6c3204c66dcd87b604927b115b27c00565d3d43f05ba2f3a2cb0d" } } }, "nbformat": 4, "nbformat_minor": 2 }