{"id":288,"date":"2020-04-12T22:11:03","date_gmt":"2020-04-12T13:11:03","guid":{"rendered":"http:\/\/localhost:8000\/?p=288"},"modified":"2021-01-17T10:28:19","modified_gmt":"2021-01-17T01:28:19","slug":"tf-idf","status":"publish","type":"post","link":"http:\/\/localhost:8000\/2020\/04\/tf-idf.html","title":{"rendered":"TF-IDF\u306e\u304a\u52c9\u5f37"},"content":{"rendered":"<p>\u5c11\u3057\u524d\u306b\u4f1a\u793e\u306e\u52c9\u5f37\u4f1a\u3067\u767a\u8868\u3057\u305f\u8cc7\u6599\u3092\u30d6\u30ed\u30b0\u306b\u3082\u8ee2\u8a18\u3057\u3066\u304a\u304d\u307e\u3059\u3002TF-IDF\u306f\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u306e\u521d\u5fc3\u8005\u306b\u3082\u3068\u3063\u3064\u304d\u3084\u3059\u304f\u7406\u89e3\u3057\u3084\u3059\u3044\u5185\u5bb9\u3067\u3057\u305f\u3002<\/p>\n<h2>Wikipedia\u306e\u8a18\u8f09\u306b\u57fa\u3065\u304d\u624b\u8a08\u7b97<\/h2>\n<h3>\u6982\u8981<\/h3>\n<p><a href=\"https:\/\/ja.wikipedia.org\/wiki\/Tf-idf\">Wikipedia<\/a>\u306b\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u8aac\u660e\u304c\u306a\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n<blockquote>\n<p>TF-IDF\u306f\u3001\u6587\u66f8\u4e2d\u306b\u542b\u307e\u308c\u308b\u5358\u8a9e\u306e\u91cd\u8981\u5ea6\u3092\u8a55\u4fa1\u3059\u308b\u624b\u6cd5\u306e1\u3064\u3067\u3042\u308a\u3001\u4e3b\u306b\u60c5\u5831\u691c\u7d22\u3084\u30c8\u30d4\u30c3\u30af\u5206\u6790\u306a\u3069\u306e\u5206\u91ce\u3067\u7528\u3044\u3089\u308c\u3066\u3044\u308b\u3002<br \/>\nTF\uff08\u82f1: Term Frequency\u3001\u5358\u8a9e\u306e\u51fa\u73fe\u983b\u5ea6\uff09\u3068<br \/>\nIDF\uff08\u82f1: Inverse Document Frequency\u3001\u9006\u6587\u66f8\u983b\u5ea6\uff09<br \/>\n\u306e\u4e8c\u3064\u306e\u6307\u6a19\u306b\u57fa\u3065\u3044\u3066\u8a08\u7b97\u3055\u308c\u308b\u3002<\/p>\n<\/blockquote>\n<p>TF\uff08\u5358\u8a9e\u51fa\u73fe\u983b\u5ea6\uff09\u3068IDF\uff08\u9006\u6587\u66f8\u983b\u5ea6\uff09\u306e\u4e8c\u3064\u306e\u6307\u6a19\u3092\u5143\u306b\u6587\u66f8\u4e2d\u306e\u5358\u8a9e\u306e\u91cd\u8981\u5ea6\u3092\u8a55\u4fa1\u3059\u308b\u624b\u6cd5\u3067\u3042\u308b\u3053\u3068\u304c\u308f\u304b\u308a\u307e\u3059\u3002<\/p>\n<h3>Wikipedia\u306e\u8a08\u7b97\u5f0f<\/h3>\n<p>\u8a08\u7b97\u5f0f\u3092\u898b\u308b\u3068\u3001TF-IDF\u5024\u306f\u3001TF\u5024\u3068IDF\u5024\u3092\u639b\u3051\u5408\u308f\u305b\u305f\u3082\u306e\u3067\u3042\u308b\u3053\u3068\u304c\u5206\u304b\u308a\u307e\u3059\u3002<\/p>\n<p><u>TF\u5024\u306f\u6587\u66f8\u4e2d\u306e\u5358\u8a9e\u51fa\u73fe\u983b\u5ea6<\/u>\u306e\u3053\u3068\u3067\u3059\u3002\u3053\u306e\u8a08\u7b97\u5f0f\u306b\u304a\u3044\u3066\u306f\u3001\u4f8b\u3048\u3070<code>I have a pen. I have an apple.<\/code>\u306e\u4e2d\u306b\u5358\u8a9e<code>have<\/code>\u306f\u3001\u51fa\u73fe\u56de\u65702\u3092\u5168\u5358\u8a9e\u65708\u3067\u5272\u3063\u305f 2\/8=0.25\u306b\u306a\u308a\u307e\u3059\u3002\u6587\u66f8\u4e2d\u306b\u983b\u7e41\u306b\u767b\u5834\u3059\u308b\u5358\u8a9e\u306f\u3053\u306e\u5024\u304c\u5927\u304d\u304f\u306a\u308a\u91cd\u8981\u3068\u5224\u65ad\u3055\u308c\u307e\u3059\u3002<\/p>\n<p><u>IDF\u5024\u306f\u3042\u308b\u5358\u8a9e\u304c\u5168\u6587\u66f8\u4e2d\u3044\u304f\u3064\u306e\u6587\u66f8\u306b\u767b\u5834\u3059\u308b\u304b\uff08\uff1d\u6587\u66f8\u983b\u5ea6\uff09\u306e\u9006\u6570<\/u>\u3067\u3059\u3002<code>a<\/code>\u3084<code>I<\/code>\u306a\u3069\u4e00\u822c\u7684\u306b\u767b\u5834\u3059\u308b\u3088\u3046\u306a\u5358\u8a9e\u306f\u3001\u6587\u66f8\u983b\u5ea6\u304c\u9ad8\u304f\u306a\u308b\u306e\u3067\u3001\u305d\u306e\u9006\u6570\u3067\u3042\u308b\u9006\u6587\u66f8\u983b\u5ea6\u306f\u5f53\u7136\u5c0f\u3055\u304f\u306a\u308a\u307e\u3059\u3002\u3064\u307e\u308a\u3001\u4e00\u822c\u7684\u306b\u767b\u5834\u3059\u308b\u3088\u3046\u306a\u5358\u8a9e\u306f\u3053\u306e\u6570\u5024\u304c\u5c0f\u3055\u304f\u306a\u308a\u3001\u91cd\u8981\u3067\u306f\u306a\u3044\u3068\u5224\u65ad\u3055\u308c\u307e\u3059\u3002IDF\u306f\u305d\u306e\u6bcd\u96c6\u56e3\u306b\u304a\u3051\u308b\u4e00\u822c\u8a9e\u3092\u9664\u5916\u3059\u308b\u4e00\u822c\u8a9e\u30d5\u30a3\u30eb\u30bf\u30fc\u3068\u3057\u3066\u50cd\u304f\u3053\u3068\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-katex\">tfidf_{i,j} = tf_{i,j} * idf_{i} = \\frac{n_{i,j}}{\\sum_{k}n_{k,j}} * log\\frac{\\vert D \\vert}{\\vert \\{ d: d \\ni t_{i} \\}\\vert}\\\\\n\u3000\u3000n_{i,j} : \u6587\u66f8j\u4e2d\u306e\u5358\u8a9ei\u306e\u51fa\u73fe\u56de\u6570\\\\\n\u3000\u3000\\vert D \\vert : \u7dcf\u6587\u66f8\u6570\\\\\n\u3000\u3000\\vert \\{ d: d \\ni t_{i} \\}\\vert : \u5358\u8a9ei\u304c\u542b\u307e\u308c\u308b\u6587\u66f8\u6570<\/code><\/pre>\n<h3>\u624b\u8a08\u7b97\u3057\u3066\u307f\u308b<\/h3>\n<pre><code>\u6587\u66f81: I have a red pen and a blue pen\n\u6587\u66f82: I like red\n\u6587\u66f83: You have a pen\n\u6587\u66f84: I have a red mechanical pencil<\/code><\/pre>\n<p>\u305d\u308c\u3067\u306f\u3001\u4e0a\u8a18\u306e4\u6587\u66f8\u306b\u304a\u3044\u3066\u3001TF-IDF\u5024\u3092\u5b9f\u969b\u306b\u8a08\u7b97\u3057\u3066\u307f\u307e\u3059\u3002\u5168\u90e8\u8a08\u7b97\u3059\u308b\u306e\u306f\u5927\u5909\u306a\u306e\u3067\u30013\u5358\u8a9e\u306b\u3064\u3044\u3066\u8a08\u7b97\u3057\u3066\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-katex\">\u6587\u66f81\u306eblue: tfidf_{blue,1} = \\frac{1}{9} * log\\frac{4}{1} = 0.066\\\\\n\u3000\u6587\u66f81\u306ered: tfidf_{red,1} = \\frac{1}{9} * log\\frac{4}{3} = 0.013\\\\\n\u3000\u6587\u66f82\u306ered: tfidf_{red,2} = \\frac{1}{3} * log\\frac{4}{3} = 0.041<\/code><\/pre>\n<h2>gensim\u306eTfidfModel\u3092\u4f7f\u3063\u3066\u5b9f\u88c5<\/h2>\n<h3>python\u5b9f\u88c5<\/h3>\n<p>gensim\u306e<code>TfidfModel<\/code>\u3092\u4f7f\u3063\u3066\u5b9f\u88c5\u3057\u307e\u3057\u305f\u3002\u6a2a\u30b9\u30af\u30ed\u30fc\u30eb\u3057\u3066\u307f\u306b\u304f\u3044\u3067\u3059\u304c\u3001\u5b9f\u88c5\u81ea\u4f53\u306f\u975e\u5e38\u306b\u30b7\u30f3\u30d7\u30eb\u3067\u3059\u306d\u3002<\/p>\n<pre><code class=\"language-python\">from typing import List, Tuple\nimport string\nimport decimal\nfrom decimal import Decimal\n\nimport nltk\nfrom nltk import tokenize\nfrom nltk.stem.porter import PorterStemmer\nfrom nltk.corpus import stopwords\nfrom gensim import corpora\nfrom gensim import models\nfrom gensim.interfaces import TransformedCorpus\n\nnltk.download(&#039;punkt&#039;)\nnltk.download(&#039;stopwords&#039;)\n\ndef tfidf(sentences: List[str]) -&gt; TransformedCorpus:\n    # \u5358\u8a9e\u5206\u5272\n    words_list: List[List[str]] = list(tokenize.word_tokenize(sentence) for sentence in sentences)\n    # \u5c0f\u6587\u5b57\u5316\n    words_list = list(list(word.lower() for word in words) for words in words_list)\n    # \u8f9e\u66f8\u4f5c\u6210\uff08\u5358\u8a9e\u6587\u5b57\u5217 -&gt; \u5358\u8a9eID\uff09\n    dictionary: corpora.Dictionary = corpora.Dictionary(words_list)\n    # \u30b3\u30fc\u30d1\u30b9\u5316\n    corpus: List[List[Tuple[int, int]]] = list(map(lambda words: dictionary.doc2bow(words), words_list))\n    # TF-IDF \u30e2\u30c7\u30eb\u751f\u6210\n    tfidf_model: models.TfidfModel = models.TfidfModel(corpus)\n    # \u30e2\u30c7\u30eb\u9069\u7528\n    tfidf_corpus: TransformedCorpus = tfidf_model[corpus]\n    return tfidf_corpus\n\nsentences: List[str] = [\n    &#039;I have a red pen and a blue pen&#039;,\n    &#039;I like red&#039;,\n    &#039;You have a pen&#039;, \n    &#039;I have a red mechanical pencil&#039;,\n]\ntfidf(sentences)<\/code><\/pre>\n<p>\u5b9f\u88c5\u5185\u5bb9\u3092\u9806\u306b\u89e3\u8aac\u3057\u3066\u307f\u307e\u3059\u3002<\/p>\n<ul>\n<li>\u5358\u8a9e\u5206\u5272\n<ul>\n<li>nlkt\u306e<code>tokenize<\/code>\u3092\u4f7f\u3063\u3066\u6587\u66f8\u3092\u5206\u5272\u3057\u3066\u5358\u8a9e\u306e\u30ea\u30b9\u30c8\u306b\u3057\u3066\u3044\u307e\u3059<\/li>\n<\/ul>\n<\/li>\n<li>\u8f9e\u66f8\u4f5c\u6210\n<ul>\n<li>\u51fa\u73fe\u3059\u308b\u5358\u8a9e\u306b\u5bfe\u3057\u3066\u3001ID\uff08\u9023\u756a\uff09\u3092\u632f\u3063\u3066\u3044\u307e\u3059<\/li>\n<\/ul>\n<\/li>\n<li>\u30b3\u30fc\u30d1\u30b9\u5316\n<ul>\n<li>\u30b3\u30fc\u30d1\u30b9\u3068\u306f\u3001\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u3092\u3057\u3084\u3059\u3044\u3088\u3046\u306b\u6587\u66f8\u3092\u69cb\u9020\u5316\u3057\u305f\u3082\u306e\u3067\u3001\u591a\u304f\u306f\u30d9\u30af\u30c8\u30eb\u8868\u73fe\u306e\u3053\u3068\u3092\u3055\u3059\u3088\u3046\u3067\u3059<\/li>\n<li><code>doc2bow<\/code>\u3092\u4f7f\u3063\u3066BoW\uff08Bag of Words\uff09\u5316\u3057\u3066\u3044\u307e\u3059\u3002Bag of Words\u306f\u76f4\u8a33\u3059\u308b\u3068\u5358\u8a9e\u306e\u888b\u3067\u3059\u3002\u540c\u4e00\u306e\u5358\u8a9e\u3092\u888b\u306b\u307e\u3068\u3081\u3066\u3001\u5358\u8a9e\u6bce\u306e\u51fa\u73fe\u56de\u6570\u3092\u6570\u3048\u308b\u3088\u3046\u306a\u30a4\u30e1\u30fc\u30b8\u3067\u6349\u3048\u3066\u3044\u307e\u3059<\/li>\n<\/ul>\n<\/li>\n<li>TF-IDF\u30e2\u30c7\u30eb\u751f\u6210\n<ul>\n<li>\u3053\u3053\u3067\u306f\u5b9f\u969b\u306e\u8a08\u7b97\u306f\u304a\u3053\u306a\u308f\u308c\u307e\u305b\u3093\u3002\u3042\u304f\u307e\u3067\u3001TF-IDF\u306e\u8a08\u7b97\u306b\u5fc5\u8981\u306a\u5909\u6570\u3092\u88cf\u3067\u4fdd\u6301\u3057\u3066\u304a\u304f\u3053\u3068\u304c\u76ee\u7684\u3067\u3059\u3002\u5177\u4f53\u7684\u306b\u306f\u3001<code>\u5168\u6587\u66f8\u6570<\/code>\u3084<code>\u3042\u308b\u5358\u8a9e\u304c\u51fa\u73fe\u3059\u308b\u6587\u66f8\u6570<\/code>\u306a\u3069\u3060\u3068\u601d\u3044\u307e\u3059<\/li>\n<\/ul>\n<\/li>\n<li>\u30e2\u30c7\u30eb\u9069\u7528\n<ul>\n<li>\u30e2\u30c7\u30eb\u306b\u5bfe\u3057\u3066TF-IDF\u5024\u3092\u8a08\u7b97\u3057\u305f\u3044\u6587\u66f8\u30d9\u30af\u30c8\u30eb\u3092\u4e0e\u3048\u308b\u3053\u3068\u3067\u3001TF-IDF\u5024\u3092\u8a08\u7b97\u3057\u307e\u3059<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3>\u8a08\u7b97\u7d50\u679c<\/h3>\n<p>\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u8a08\u7b97\u7d50\u679c\u3068\u306a\u308a\u307e\u3057\u305f\u3002<\/p>\n<pre><code class=\"language-python\">[(&#039;a&#039;, 0.2284), (&#039;and&#039;, 0.5504), (&#039;blue&#039;, 0.5504), (&#039;have&#039;, 0.1142), (&#039;i&#039;, 0.1142), (&#039;pen&#039;, 0.5504), (&#039;red&#039;, 0.1142)]\n[(&#039;i&#039;, 0.1991), (&#039;red&#039;, 0.1991), (&#039;like&#039;, 0.9595)]\n[(&#039;a&#039;, 0.1795), (&#039;have&#039;, 0.1795), (&#039;pen&#039;, 0.4326), (&#039;you&#039;, 0.8651)]\n[(&#039;a&#039;, 0.1408), (&#039;have&#039;, 0.1408), (&#039;i&#039;, 0.1408), (&#039;red&#039;, 0.1408), (&#039;mechanical&#039;, 0.6785), (&#039;pencil&#039;, 0.6785)]<\/code><\/pre>\n<p><code>and<\/code> <code>you<\/code> <code>blue<\/code> <code>pen<\/code> <code>like<\/code> <code>pencil<\/code> <code>mechanical<\/code>\u306a\u3069\u304c0.5\u3092\u8d85\u3048\u308b\u5024\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002<code>and<\/code> <code>you<\/code>\u306e\u3088\u3046\u306a\u4e00\u822c\u7684\u306a\u5358\u8a9e\u304c\u9ad8\u3044\u5024\u306b\u306a\u3063\u3066\u3044\u308b\u306e\u3067\u9055\u548c\u611f\u304c\u3042\u308a\u307e\u3059\u304c\u3001\u8a08\u7b97\u5f0f\u3092\u8003\u3048\u308b\u3068\u91cd\u8981\u5ea6\u304c\u9ad8\u3044\u3082\u306e\u304c\u5024\u304c\u9ad8\u304f\u306a\u3063\u3066\u3044\u308b\u5370\u8c61\u306f\u3042\u308a\u307e\u3059\u3002<\/p>\n<h2>Wikipedia\u3068gensim\u3067\u8a08\u7b97\u7d50\u679c\u304c\u7570\u306a\u308b\u7406\u7531<\/h2>\n<h3>\u6bd4\u8f03\u7d50\u679c<\/h3>\n<p>\u6b8b\u5ff5\u306a\u304c\u3089\u3001Wikipedia\u3092\u5143\u306b\u3057\u305f\u624b\u8a08\u7b97\u3068gensim\u306e<code>TfidfModel<\/code>\u3067\u306e\u8a08\u7b97\u7d50\u679c\u306f\u6570\u5024\u306e\u50be\u5411\u306f\u4f3c\u3066\u3044\u308b\u3088\u3046\u3067\u3059\u304c\u7570\u306a\u308a\u307e\u3059\u3002\u3053\u308c\u306fgensim\u306e\u30c7\u30d5\u30a9\u30eb\u30c8\u306eTF-IDF\u306e\u8a08\u7b97\u5f0f\u304cWikipedia\u306e\u8a08\u7b97\u5f0f\u3068\u7570\u306a\u308b\u305f\u3081\u3067\u3059\u3002<\/p>\n<pre><code class=\"language-katex\">Wikipedia:\\\\\n\u3000\u3000tfidf_{blue,1} = 0.066\\\\\n\u3000\u3000tfidf_{red,1} = 0.013\\\\\n\u3000\u3000tfidf_{red,2} = 0.041\\\\\n\u3000gensim:\\\\\n\u3000\u3000tfidf_{blue,1} = 0.550\\\\\n\u3000\u3000tfidf_{red,1} = 0.114\\\\\n\u3000\u3000tfidf_{red,2} = 0.199<\/code><\/pre>\n<h3>gensim\u306e\u8a08\u7b97\u5f0f<\/h3>\n<p><a href=\"https:\/\/radimrehurek.com\/gensim\/models\/tfidfmodel.html\">gensim\u306eTfidfModel\u306e\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8<\/a>\u3092\u898b\u308b\u3068\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u3053\u3068\u304c\u66f8\u3044\u3066\u3042\u308a\u307e\u3059\uff08\u9069\u5f53\u306b\u610f\u8a33\u3057\u3066\u3042\u308a\u307e\u3059\uff09\u3002<\/p>\n<blockquote>\n<p>TF-IDF\u306e\u8a08\u7b97\u306f\u3001\u5358\u8a9e\u983b\u5ea6\u3068\u9006\u6587\u66f8\u983b\u5ea6\u3092\u639b\u3051\u305f\u7d50\u679c\u306e\u6587\u66f8\u30d9\u30af\u30c8\u30eb\u306e\u9577\u3055\u3092\u5358\u4f4d\u9577\u306b\u30ce\u30fc\u30de\u30e9\u30a4\u30ba\u3059\u308b\u3053\u3068\u3067\u7b97\u51fa\u3057\u3066\u304a\u308a\u3001\u6bcd\u96c6\u56e3\u6587\u66f8\u7fa4D\u306e\u30b3\u30fc\u30d1\u30b9\u306b\u304a\u3051\u308b\u6587\u66f8j\u306b\u542b\u307e\u308c\u308b\u5358\u8a9ei\u306e\u6b63\u898f\u5316\u524d\u306e\u91cd\u307f\u306f\u4ee5\u4e0b\u306e\u8a08\u7b97\u5f0f\u3067\u8868\u3055\u308c\u308b\u3002<\/p>\n<\/blockquote>\n<pre><code class=\"language-katex\">weight_{i,j} = frequency_{i,j} * log_{2}\\frac{D}{document\\_freq_{i}}<\/code><\/pre>\n<p>\u3053\u306e\u4e2d\u306b\u306f\u3001\u5177\u4f53\u7684\u306a\u8a08\u7b97\u5f0f\u304c\u66f8\u3044\u3066\u306a\u3044\u3067\u3059\u304c\u3001\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u3092\u5c11\u3057\u773a\u3081\u3066\u3044\u308b\u3068\u3001<code>smartirs<\/code>\u3068\u3044\u3046\u30aa\u30d7\u30b7\u30e7\u30f3\u304c\u3042\u308a\u3001\u3053\u306e\u30aa\u30d7\u30b7\u30e7\u30f3\u3067<code>TF<\/code> <code>IDF<\/code> <code>\u6587\u66f8length\u306e\u6b63\u898f\u5316<\/code> \u306e\u4e09\u3064\u306e\u8a08\u7b97\u65b9\u5f0f\u3092\u6307\u5b9a\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/p>\n<p>\u30c7\u30d5\u30a9\u30eb\u30c8\u3060\u3068<code>nfc<\/code>\u306a\u306e\u3067\u3001\u5177\u4f53\u7684\u306a\u8a08\u7b97\u5f0f\u306f<a href=\"https:\/\/en.wikipedia.org\/wiki\/SMART_Information_Retrieval_System\">\u3053\u3061\u3089<\/a>\u3092\u53c2\u7167\u3057\u3066\u78ba\u8a8d\u3067\u304d\u307e\u3059\u3002Wikipedia\u306e\u8a18\u8ff0\u306b\u5408\u308f\u305b\u3066\u5909\u6570\u540d\u3092\u5c11\u3057\u8aad\u307f\u66ff\u3048\u3066\u8aac\u660e\u3057\u3066\u3044\u3053\u3046\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n<h4>TF<\/h4>\n<pre><code class=\"language-katex\">f_{i,j}: \u6587\u66f8j\u4e2d\u306e\u5358\u8a9ei\u306e\u51fa\u73fe\u56de\u6570<\/code><\/pre>\n<p>Wikipedia\u3067\u306f\u6587\u66f8j\u306e\u5358\u8a9e\u6570\u3067\u5272\u308b\u3053\u3068\u3067\u6587\u66f8length\u306e\u9055\u3044\u3092\u6b63\u898f\u5316\u3057\u3066\u3044\u307e\u3057\u305f\u304c\u3001\u3053\u3061\u3089\u3067\u306f\u5272\u3063\u3066\u3044\u307e\u305b\u3093\u3002\u6587\u66f8length\u306e\u6b63\u898f\u5316\u3092\u5225\u3067\u884c\u306a\u3063\u3066\u3044\u308b\u305f\u3081\u3067\u3059\u306d\u3002<\/p>\n<h4>IDF<\/h4>\n<pre><code class=\"language-katex\">log_{2}\\frac{D}{\\vert \\{ d: d \\ni t_{i} \\}\\vert}:\\\\\n\u3000\u3000\u5358\u8a9ei\u304c\u542b\u307e\u308c\u308b\u6587\u66f8\u6570\u3092\u7dcf\u6587\u66f8\u6570\u3067\u5272\u3063\u305f\u5024\uff08\u6587\u66f8\u983b\u5ea6\uff09\u306e\u9006\u6570\u306e\u5bfe\u6570\u3092\u3068\u3063\u305f\u3082\u306e<\/code><\/pre>\n<p>Wikipedia\u306f\u3001\u5e9510\u3067\u5bfe\u6570\u3092\u3068\u3063\u3066\u3044\u307e\u3057\u305f\u304c\u3001\u3053\u3061\u3089\u3067\u306f\u5e952\u3067\u5bfe\u6570\u3092\u3068\u3063\u3066\u3044\u308b\u70b9\u304c\u7570\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n<h4>\u6587\u66f8length\u306e\u6b63\u898f\u5316<\/h4>\n<pre><code class=\"language-katex\">\\sqrt{\\sum^{t}_{i=1}w^{2}_{i,j}}:\\\\\n\u3000\u3000\u6587\u66f8j\u306b\u542b\u307e\u308c\u308b\u5168\u5358\u8a9e\u306e\u91cd\u307f\u306e\u4e8c\u4e57\u306e\u5408\u8a08\u306e\u5e73\u65b9\u6839<\/code><\/pre>\n<p><code>\u6587\u66f8j\u306b\u542b\u307e\u308c\u308b\u5168\u5358\u8a9e\u306e\u91cd\u307f\u306e\u4e8c\u4e57\u306e\u5408\u8a08\u306e\u5e73\u65b9\u6839<\/code>\u3068\u306f\u3001\u6587\u66f8j\u306e\u91cd\u307f\u30d9\u30af\u30c8\u30eb\u306e\u30c8\u30eb\u30af\u3001\u3064\u307e\u308a\u3001\u30d9\u30af\u30c8\u30eb\u306e\u5927\u304d\u3055\u3092\u8868\u3057\u3066\u3044\u307e\u3059\u3002<br \/>\nTF\u304a\u3088\u3073IDF\u3067\u8a08\u7b97\u3057\u305f\u6587\u66f8j\u4e2d\u306e\u5404\u5358\u8a9e\u306e\u91cd\u307f\u3092\u30c8\u30eb\u30af\u3067\u5272\u308b\u3053\u3068\u306b\u3088\u3063\u3066\u3001\u6587\u66f8\u306e\u91cd\u307f\u30d9\u30af\u30c8\u30eb\u306e\u5927\u304d\u3055\u3092<code>1<\/code>\u306b\u6b63\u898f\u5316\u3057\u3066\u3044\u307e\u3059\u3002\u751f\u6210\u3057\u305f\u30d9\u30af\u30c8\u30eb\u306e\u30e6\u30fc\u30af\u30ea\u30c3\u30c9\u8ddd\u96e2\u3092\u3068\u3063\u3066\u985e\u4f3c\u5ea6\u3092\u6bd4\u8f03\u3057\u305f\u308a\u3059\u308b\u306e\u3067\u3042\u308c\u3070\u3001\u3053\u306e\u6b63\u898f\u5316\u306f\u5fc5\u9808\u3067\u3059\u306d\u3002\u9006\u306b\u30b3\u30b5\u30a4\u30f3\u985e\u4f3c\u5ea6\u3067\u6bd4\u8f03\u3059\u308b\u306e\u3067\u3042\u308c\u3070\u3001\u3053\u306e\u6b63\u898f\u5316\u306f\u4e0d\u8981\u305d\u3046\u3067\u3059\u3002<\/p>\n<h4>\u6700\u7d42\u7684\u306a\u8a08\u7b97\u5f0f<\/h4>\n<p>\u3053\u306e\u4e09\u3064\u306e\u8a08\u7b97\u5f0f\u3092\u307e\u3068\u3081\u305f\u6700\u7d42\u7684\u306a\u8a08\u7b97\u5f0f\u306f\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-katex\">tfidf_{i,j} = \\frac{f_{i,j} * log_{2}\\frac{D}{\\vert \\{ d: d \\ni t_{i} \\}\\vert}}{\\sqrt{\\sum^{t}_{i=1}w^{2}_{i,j}}}<\/code><\/pre>\n<h3>\u624b\u8a08\u7b97\u3057\u3066\u307f\u308b<\/h3>\n<p>\u3053\u306e\u8a08\u7b97\u5f0f\u306b\u3057\u305f\u304c\u3063\u3066\u6587\u66f82<code>I like red<\/code>\u306b\u3064\u3044\u3066\u8a08\u7b97\u3057\u305f\u7d50\u679c\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-katex\">w_{I,2} = 1 * log_{2}\\frac{4}{3} = 0.415\\\\\n\u3000w_{like,2} = 1 * log_{2}\\frac{4}{1} = 2.000\\\\\n\u3000w_{red,2} = 1 * log_{2}\\frac{4}{3} = 0.415\\\\\n\u3000norm_{2} = \\sqrt{0.415^{2} + 2.000^{2} + 0.415^{2}} = 2.084\\\\\n\u3000\\\\\n\u3000tfidf_{I,2} = \\frac{w_{I,2}}{norm_{2}} = \\frac{0.415}{2.084} = 0.1991\\\\\n\u3000tfidf_{like,2} = \\frac{w_{like,2}}{norm_{2}} = \\frac{2.000}{2.084} = 0.9595\\\\\n\u3000tfidf_{red,2} = \\frac{w_{red,2}}{norm_{2}} = \\frac{0.415}{2.084} = 0.1991<\/code><\/pre>\n<p>\u4e00\u65b9\u3067\u3001gensim\u3092\u3064\u304b\u3063\u305f\u30d7\u30ed\u30b0\u30e9\u30e0\u306e\u7d50\u679c\u306f\u4ee5\u4e0b\u306e\u901a\u308a\u3067\u3059\u3002<\/p>\n<pre><code class=\"language-python\">[(&#039;i&#039;, 0.1991), (&#039;red&#039;, 0.1991), (&#039;like&#039;, 0.9595)]<\/code><\/pre>\n<p>\u5b8c\u5168\u4e00\u81f4\u3057\u307e\u3057\u305f\u306d\uff01<\/p>\n<h2>[\u86c7\u8db3] stopword\u3092\u9664\u5916\u3059\u308b\u3053\u3068\u3067\u91cd\u307f\u4ed8\u3051\u306e\u7d50\u679c\u3092\u6539\u5584<\/h2>\n<p>\u672c\u7b4b\u3068\u306f\u95a2\u4fc2\u306a\u3044\u3044\u3067\u3059\u304c\u3001\u4e0a\u8a18\u3067\u8a18\u8f09\u3057\u305f\u5b9f\u88c5\u3067\u306f<code>and<\/code> <code>you<\/code>\u306a\u3069\u304c\u91cd\u8981\u306a\u5358\u8a9e\u3068\u3057\u3066\u30d4\u30c3\u30af\u30a2\u30c3\u30d7\u3055\u308c\u3066\u3044\u3066\u9055\u548c\u611f\u304c\u3042\u308a\u307e\u3057\u305f\u3002\u3053\u308c\u306b\u3064\u3044\u3066\u306f\u3001\u524d\u51e6\u7406\u3067stopword\uff08\u4e00\u822c\u5f8c\u306a\u3069\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u3067\u30ce\u30a4\u30ba\u3068\u306a\u308b\u3088\u3046\u306a\u5358\u8a9e\uff09\u3092\u9664\u5916\u3059\u308b\u51e6\u7406\u3092\u8ffd\u52a0\u3059\u308b\u3053\u3068\u3067\u6539\u5584\u6539\u5584\u3067\u304d\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-python\">    (\u7701\u7565)\n    # \u5c0f\u6587\u5b57\u5316\n    words_list = list(list(word.lower() for word in words) for words in words_list)\n    # \u30b9\u30c8\u30c3\u30d7\u30ef\u30fc\u30c9\u3001\u8a18\u53f7\u30011\u6587\u5b57\u306e\u82f1\u6570\u5b57\u3092\u9664\u5916\n    stop_words: List[str] = stopwords.words(&#039;english&#039;)\n    exclude_words: List[str] = stop_words + list(string.ascii_lowercase) + list(string.digits) + list(string.punctuation)\n    words_list: List[List[str]] = list(\n        list(word for word in words if word not in exclude_words) for words in words_list\n    )\n    # \u8f9e\u66f8\u4f5c\u6210\uff08\u5358\u8a9e\u6587\u5b57\u5217 -&gt; \u5358\u8a9eID\uff09\n    dictionary: corpora.Dictionary = corpora.Dictionary(words_list)\n    (\u7701\u7565)<\/code><\/pre>\n<p>\u3053\u308c\u306b\u3088\u308a\u51fa\u529b\u7d50\u679c\u304c\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u5909\u5316\u3057\u307e\u3059\u3002<\/p>\n<pre><code class=\"language-python\"># input\u306e\u6587\u66f8\n&quot;I have a red pen and a blue pen&quot;\n&quot;I like red&quot;\n&quot;You have a pen&quot;\n&quot;I have a red mechanical pencil&quot;\n# \u5909\u66f4\u524d\uff08stopword\u9664\u5916\u306a\u3057\uff09\n[(&#039;a&#039;, 0.2284), (&#039;and&#039;, 0.5504), (&#039;blue&#039;, 0.5504), (&#039;have&#039;, 0.1142), (&#039;i&#039;, 0.1142), (&#039;pen&#039;, 0.5504), (&#039;red&#039;, 0.1142)]\n[(&#039;i&#039;, 0.1991), (&#039;red&#039;, 0.1991), (&#039;like&#039;, 0.9595)]\n[(&#039;a&#039;, 0.1795), (&#039;have&#039;, 0.1795), (&#039;pen&#039;, 0.4326), (&#039;you&#039;, 0.8651)]\n[(&#039;a&#039;, 0.1408), (&#039;have&#039;, 0.1408), (&#039;i&#039;, 0.1408), (&#039;red&#039;, 0.1408), (&#039;mechanical&#039;, 0.6785), (&#039;pencil&#039;, 0.6785)]\n# \u5909\u66f4\u5f8c\uff08stopword\u9664\u5916\u3042\u308a\uff09\n[(&#039;blue&#039;, 0.6996), (&#039;pen&#039;, 0.6996), (&#039;red&#039;, 0.1452)]\n[(&#039;red&#039;, 0.2032), (&#039;like&#039;, 0.9791)]\n[(&#039;pen&#039;, 1.0)]\n[(&#039;red&#039;, 0.1452), (&#039;mechanical&#039;, 0.6996), (&#039;pencil&#039;, 0.6996)]<\/code><\/pre>\n<p>\u3053\u308c\u3067\u9055\u548c\u611f\u304c\u306a\u304f\u306a\u308a\u307e\u3057\u305f\u306d\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u5c11\u3057\u524d\u306b\u4f1a\u793e\u306e\u52c9\u5f37\u4f1a\u3067\u767a\u8868\u3057\u305f\u8cc7\u6599\u3092\u30d6\u30ed\u30b0\u306b\u3082\u8ee2\u8a18\u3057\u3066\u304a\u304d\u307e\u3059\u3002TF-IDF\u306f\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u306e\u521d\u5fc3\u8005\u306b\u3082\u3068\u3063\u3064\u304d\u3084\u3059\u304f\u7406\u89e3\u3057\u3084\u3059\u3044\u5185\u5bb9\u3067\u3057\u305f\u3002 Wikipedia\u306e\u8a18\u8f09\u306b\u57fa\u3065\u304d\u624b\u8a08\u7b97 \u6982\u8981 Wikipedia\u306b\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u8aac\u660e\u304c\u306a\u3055\u308c\u3066\u3044\u307e\u3059\u3002 TF-IDF\u306f\u3001\u6587\u66f8\u4e2d\u306b\u542b\u307e\u308c\u308b\u5358\u8a9e\u306e\u91cd\u8981\u5ea6\u3092\u8a55\u4fa1\u3059\u308b\u624b\u6cd5\u306e1\u3064\u3067\u3042\u308a\u3001\u4e3b\u306b\u60c5\u5831\u691c\u7d22\u3084\u30c8\u30d4\u30c3\u30af\u5206\u6790\u306a\u3069\u306e\u5206\u91ce\u3067\u7528\u3044\u3089\u308c\u3066\u3044\u308b\u3002 TF\uff08\u82f1: Term Frequency\u3001\u5358\u8a9e\u306e\u51fa\u73fe\u983b\u5ea6\uff09\u3068 IDF\uff08\u82f1: Inverse Document Frequency\u3001\u9006\u6587\u66f8\u983b\u5ea6\uff09 \u306e\u4e8c\u3064\u306e\u6307\u6a19\u306b\u57fa\u3065\u3044\u3066\u8a08\u7b97\u3055\u308c\u308b\u3002 TF\uff08\u5358\u8a9e\u51fa\u73fe\u983b\u5ea6\uff09\u3068IDF\uff08\u9006\u6587\u66f8\u983b\u5ea6\uff09\u306e\u4e8c\u3064\u306e\u6307\u6a19\u3092\u5143\u306b\u6587\u66f8\u4e2d\u306e\u5358\u8a9e\u306e\u91cd\u8981\u5ea6\u3092\u8a55\u4fa1\u3059\u308b\u624b\u6cd5\u3067\u3042\u308b\u3053\u3068\u304c\u308f\u304b\u308a\u307e\u3059\u3002 Wikipedia\u306e\u8a08\u7b97\u5f0f \u8a08\u7b97\u5f0f\u3092\u898b\u308b\u3068\u3001TF-IDF\u5024\u306f\u3001TF\u5024\u3068IDF\u5024\u3092\u639b\u3051\u5408\u308f\u305b\u305f\u3082\u306e\u3067\u3042\u308b\u3053\u3068\u304c\u5206\u304b\u308a\u307e\u3059\u3002 TF <a href=\"http:\/\/localhost:8000\/2020\/04\/tf-idf.html\" class=\"read-more button-fancy -red\"><span class=\"btn-arrow\"><\/span><span class=\"twp-read-more text\">Continue Reading<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[30],"tags":[],"_links":{"self":[{"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/posts\/288"}],"collection":[{"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/comments?post=288"}],"version-history":[{"count":1,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/posts\/288\/revisions"}],"predecessor-version":[{"id":289,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/posts\/288\/revisions\/289"}],"wp:attachment":[{"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/media?parent=288"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/categories?post=288"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/localhost:8000\/wp-json\/wp\/v2\/tags?post=288"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}