mllib建立推薦模型

數據準備

數據包含在ml-100k的文件夾中，文件夾中比較重要的幾個文件是u.user（用戶屬性數據集）、u.item（電影元數據）和u.data（用戶對電影的評分數據）
(1)u.user數據的每列分別代表用戶ID、年齡、性別、職業、郵政編碼，其分隔符爲“|”；
(2)u.item數據包含的列爲電影ID、電影名、上映日期及其它一些屬性信息，分隔符也爲“|”；
(3)u.data數據包含用戶ID、電影ID、評分（1-5分）和時間戳，分隔符爲製表符（\t）
其他數據的說明可以從README獲取

rawData = sc.textFile('hdfs://master:9000/ml-100k/u.data')
rawData.take(5)

[u'196\t242\t3\t881250949',
 u'186\t302\t3\t891717742',
 u'22\t377\t1\t878887116',
 u'244\t51\t2\t880606923',
 u'166\t346\t1\t886397596']

rawRatings = rawData.map(lambda line: line.split('\t'))
rawRatings.take(5)

[[u'196', u'242', u'3', u'881250949'],
 [u'186', u'302', u'3', u'891717742'],
 [u'22', u'377', u'1', u'878887116'],
 [u'244', u'51', u'2', u'880606923'],
 [u'166', u'346', u'1', u'886397596']]

在mllib的recommendation模塊中，提供了一個類Rating，用於將數據轉化爲用於ALS算法的指定結構，轉化的過程如下：

from pyspark.mllib.recommendation import Rating
ratings = rawRatings.map(lambda line: Rating(int(line[0]),
                                             int(line[1]),
                                             float(line[2])
                                            )
                        )
ratings.take(5)

[Rating(user=196, product=242, rating=3.0),
 Rating(user=186, product=302, rating=3.0),
 Rating(user=22, product=377, rating=1.0),
 Rating(user=244, product=51, rating=2.0),
 Rating(user=166, product=346, rating=1.0)]

轉化後的RDD是由Rating對象構成的，從結果中可以看出，Rating對象包含了三個值，user、product和rating，即用戶、產品和打分。原始評分數據中第四列爲時間戳，在本例中用不到，因此被拋棄，Rating類只能接受三個參數。該對象的使用也很簡單，要提取指定的值，只需要使用原點“.”加屬性名即可，也可以使用索引，將要引用的成員的索引號以方括號跟在對象後面即可

r_1 = ratings.first()
print r_1.user, r_1.product, r_1.rating
print r_1[0], r_1[1], r_1[2]

196 242 3.0
196 242 3.0

建模

from pyspark.mllib.recommendation import ALS
cf_model = ALS.train(ratings, 50, 10, 0.01, nonnegative=False, seed=12345)
cf_model

<pyspark.mllib.recommendation.MatrixFactorizationModel at 0x7fd1a8023390>

訓練後的模型爲一個MatrixFactorizationModel對象，該對象提供的方法可用於提取因子矩陣和進行預測，比如:

cf_model.userFeatures().first()

(2,
 array('d', [0.8257154226303101, -0.08174031972885132, -0.4485216736793518, 0.2816902697086334, 0.281324565410614, 0.2871280014514923, -0.28037557005882263, -0.5780994892120361, 0.04380865767598152, -0.03685721382498741, 0.33663856983184814, 0.8575121164321899, -0.26763004064559937, -0.22665703296661377, -0.030370648950338364, -0.4087982177734375, 0.28470417857170105, 0.17012149095535278, -0.46445152163505554, -0.39363399147987366, 0.4133472442626953, 0.0196047555655241, -0.6278623342514038, 0.8203023672103882, 0.36110371351242065, -0.3623308539390564, 0.07974052429199219, 0.3489876985549927, 0.009540693834424019, -0.1018930971622467, -0.3096586763858795, -0.08348742127418518, 0.546208918094635, 0.14119906723499298, -0.11057484149932861, 0.003356723114848137, -0.42252105474472046, 0.5306751728057861, 0.18785302340984344, 0.30044302344322205, -0.017208704724907875, 0.4387732148170471, -0.06367648392915726, 0.1654045730829239, 0.28026890754699707, -0.18949449062347412, -0.17139069736003876, -0.24911031126976013, 0.05620288848876953, -0.48843708634376526]))

cf_model.productFeatures().take(1)

[(2,
  array('d', [0.7962743639945984, 0.17431776225566864, -0.1990462988615036, -1.1859782934188843, -0.3959435522556305, -0.5246215462684631, 0.5594768524169922, -1.0115996599197388, 0.16964347660541534, 0.5268467664718628, -0.25127914547920227, -0.6580895185470581, 0.5533314943313599, 0.2781536877155304, -0.8546806573867798, 0.003281824290752411, -0.1445930451154709, -0.4302116930484772, -0.9390072226524353, -0.012757998891174793, -0.2912135422229767, -0.1968940943479538, -1.0604552030563354, 0.8730130195617676, 0.20659275352954865, -1.2206825017929077, -1.2894175052642822, 0.3126126825809479, 0.3025003671646118, 0.3809513449668884, -0.6017365455627441, 0.46676522493362427, -0.17819972336292267, 0.03601028397679329, 0.5260732769966125, -0.3788948357105255, -1.3027641773223877, -0.08637615293264389, -0.22254005074501038, 1.1796964406967163, 0.7695205807685852, 0.420034259557724, -0.31719499826431274, -0.43826058506965637, 0.7229039669036865, 0.1820073425769806, 0.3955117464065552, 0.4843427240848541, 0.5735178589820862, -1.1487818956375122]))]

預測

cf_model.predict(123, 456)

0.5189201089615622

一般來說我們不會關心某個具體用戶對某部具體電影的評分，而是希望能對預測的用戶評分進行排序，此時需要使用predictAll方法，該方法接受“用戶-產品”類型的RDD，返回所有預測評分，比如我們可以先生成一個RDD，該RDD包含用戶123所有評分過的電影：

r_123 = ratings.filter(lambda r: r.user == 123)
user_product_pairs = r_123.map(lambda r: (r.user, r.product))
user_product_pairs.take(5)

[(123, 427), (123, 531), (123, 135), (123, 192), (123, 13)]

預測123所有評分

est_123 = cf_model.predictAll(user_product_pairs)
est_123.take(5)

[Rating(user=123, product=14, rating=4.781807508293159),
 Rating(user=123, product=192, rating=4.8440853953154654),
 Rating(user=123, product=64, rating=3.189367663814919),
 Rating(user=123, product=432, rating=4.826974367452291),
 Rating(user=123, product=480, rating=3.4997134540180883)]

因爲所有預測的評分都是用戶123已經確實評分過的，我們可以將兩個結果放在一起，方便比較：

left = r_123.map(lambda r: (r.product, r.rating))
right = est_123.map(lambda r: (r.product, r.rating))
left.join(right).take(5)

[(704, (3.0, 3.12740470516471)),
 (64, (3.0, 3.189367663814919)),
 (132, (3.0, 3.255692030148917)),
 (192, (5.0, 4.8440853953154654)),
 (288, (3.0, 2.6833371635987526))]

recommendProducts和recommendUsers推薦產品和用戶

topK_users = cf_model.recommendUsers(456, 5)
topK_users

[Rating(user=534, product=456, rating=4.915272594291057),
 Rating(user=620, product=456, rating=4.554502950952971),
 Rating(user=462, product=456, rating=4.548246109196231),
 Rating(user=283, product=456, rating=4.514564326007004),
 Rating(user=56, product=456, rating=4.2624704017243324)]

topK_movies = cf_model.recommendProducts(123, 5)
topK_movies

[Rating(user=123, product=287, rating=6.238110881748787),
 Rating(user=123, product=269, rating=6.168910818255823),
 Rating(user=123, product=515, rating=5.954825359796975),
 Rating(user=123, product=416, rating=5.921099652606939),
 Rating(user=123, product=503, rating=5.670748725720875)]

連接電影名稱

movies = sc.textFile('hdfs://master:9000/ml-100k/u.item')
titles = movies.map(lambda line: line.split('|'))\
               .map(lambda x: (int(x[0]), x[1])).collectAsMap()
titles

{1: u'Toy Story (1995)',
 2: u'GoldenEye (1995)',
 3: u'Four Rooms (1995)',
 4: u'Get Shorty (1995)',
 5: u'Copycat (1995)',
 6: u'Shanghai Triad (Yao a yao yao dao waipo qiao) (1995)',
 7: u'Twelve Monkeys (1995)',
 8: u'Babe (1995)',
 9: u'Dead Man Walking (1995)',
 10: u'Richard III (1995)',
 11: u'Seven (Se7en) (1995)',
 12: u'Usual Suspects, The (1995)',
 13: u'Mighty Aphrodite (1995)',
 14: u'Postino, Il (1994)',
 15: u"Mr. Holland's Opus (1995)",
 16: u'French Twist (Gazon maudit) (1995)',
 17: u'From Dusk Till Dawn (1996)',
 18: u'White Balloon, The (1995)',
 19: u"Antonia's Line (1995)",
 20: u'Angels and Insects (1995)',
 21: u'Muppet Treasure Island (1996)',
 22: u'Braveheart (1995)',
 23: u'Taxi Driver (1976)',
 24: u'Rumble in the Bronx (1995)',
 25: u'Birdcage, The (1996)',
 26: u'Brothers McMullen, The (1995)',
 27: u'Bad Boys (1995)',
 28: u'Apollo 13 (1995)',
 29: u'Batman Forever (1995)',
 30: u'Belle de jour (1967)',
 31: u'Crimson Tide (1995)',
 32: u'Crumb (1994)',
 33: u'Desperado (1995)',
 34: u'Doom Generation, The (1995)',
 35: u'Free Willy 2: The Adventure Home (1995)',
 36: u'Mad Love (1995)',
 37: u'Nadja (1994)',
 38: u'Net, The (1995)',
 39: u'Strange Days (1995)',
 40: u'To Wong Foo, Thanks for Everything! Julie Newmar (1995)',
 41: u'Billy Madison (1995)',
 42: u'Clerks (1994)',
 43: u'Disclosure (1994)',
 44: u'Dolores Claiborne (1994)',
 45: u'Eat Drink Man Woman (1994)',
 46: u'Exotica (1994)',
 47: u'Ed Wood (1994)',
 48: u'Hoop Dreams (1994)',
 49: u'I.Q. (1994)',
 50: u'Star Wars (1977)',
 51: u'Legends of the Fall (1994)',
 52: u'Madness of King George, The (1994)',
 53: u'Natural Born Killers (1994)',
 54: u'Outbreak (1995)',
 55: u'Professional, The (1994)',
 56: u'Pulp Fiction (1994)',
 57: u'Priest (1994)',
 58: u'Quiz Show (1994)',
 59: u'Three Colors: Red (1994)',
 60: u'Three Colors: Blue (1993)',
 61: u'Three Colors: White (1994)',
 62: u'Stargate (1994)',
 63: u'Santa Clause, The (1994)',
 64: u'Shawshank Redemption, The (1994)',
 65: u"What's Eating Gilbert Grape (1993)",
 66: u'While You Were Sleeping (1995)',
 67: u'Ace Ventura: Pet Detective (1994)',
 68: u'Crow, The (1994)',
 69: u'Forrest Gump (1994)',
 70: u'Four Weddings and a Funeral (1994)',
 71: u'Lion King, The (1994)',
 72: u'Mask, The (1994)',
 73: u'Maverick (1994)',
 74: u'Faster Pussycat! Kill! Kill! (1965)',
 75: u'Brother Minister: The Assassination of Malcolm X (1994)',
 76: u"Carlito's Way (1993)",
 77: u'Firm, The (1993)',
 78: u'Free Willy (1993)',
 79: u'Fugitive, The (1993)',
 80: u'Hot Shots! Part Deux (1993)',
 81: u'Hudsucker Proxy, The (1994)',
 82: u'Jurassic Park (1993)',
 83: u'Much Ado About Nothing (1993)',
 84: u"Robert A. Heinlein's The Puppet Masters (1994)",
 85: u'Ref, The (1994)',
 86: u'Remains of the Day, The (1993)',
 87: u'Searching for Bobby Fischer (1993)',
 88: u'Sleepless in Seattle (1993)',
 89: u'Blade Runner (1982)',
 90: u'So I Married an Axe Murderer (1993)',
 91: u'Nightmare Before Christmas, The (1993)',
 92: u'True Romance (1993)',
 93: u'Welcome to the Dollhouse (1995)',
 94: u'Home Alone (1990)',
 95: u'Aladdin (1992)',
 96: u'Terminator 2: Judgment Day (1991)',
 97: u'Dances with Wolves (1990)',
 98: u'Silence of the Lambs, The (1991)',
 99: u'Snow White and the Seven Dwarfs (1937)',
 100: u'Fargo (1996)',
 101: u'Heavy Metal (1981)',
 102: u'Aristocats, The (1970)',
 103: u'All Dogs Go to Heaven 2 (1996)',
 104: u'Theodore Rex (1995)',
 105: u'Sgt. Bilko (1996)',
 106: u'Diabolique (1996)',
 107: u'Moll Flanders (1996)',
 108: u'Kids in the Hall: Brain Candy (1996)',
 109: u'Mystery Science Theater 3000: The Movie (1996)',
 110: u'Operation Dumbo Drop (1995)',
 111: u'Truth About Cats & Dogs, The (1996)',
 112: u'Flipper (1996)',
 113: u'Horseman on the Roof, The (Hussard sur le toit, Le) (1995)',
 114: u'Wallace & Gromit: The Best of Aardman Animation (1996)',
 115: u'Haunted World of Edward D. Wood Jr., The (1995)',
 116: u'Cold Comfort Farm (1995)',
 117: u'Rock, The (1996)',
 118: u'Twister (1996)',
 119: u'Maya Lin: A Strong Clear Vision (1994)',
 120: u'Striptease (1996)',
 121: u'Independence Day (ID4) (1996)',
 122: u'Cable Guy, The (1996)',
 123: u'Frighteners, The (1996)',
 124: u'Lone Star (1996)',
 125: u'Phenomenon (1996)',
 126: u'Spitfire Grill, The (1996)',
 127: u'Godfather, The (1972)',
 128: u'Supercop (1992)',
 129: u'Bound (1996)',
 130: u'Kansas City (1996)',
 131: u"Breakfast at Tiffany's (1961)",
 132: u'Wizard of Oz, The (1939)',
 133: u'Gone with the Wind (1939)',
 134: u'Citizen Kane (1941)',
 135: u'2001: A Space Odyssey (1968)',
 136: u'Mr. Smith Goes to Washington (1939)',
 137: u'Big Night (1996)',
 138: u'D3: The Mighty Ducks (1996)',
 139: u'Love Bug, The (1969)',
 140: u'Homeward Bound: The Incredible Journey (1993)',
 141: u'20,000 Leagues Under the Sea (1954)',
 142: u'Bedknobs and Broomsticks (1971)',
 143: u'Sound of Music, The (1965)',
 144: u'Die Hard (1988)',
 145: u'Lawnmower Man, The (1992)',
 146: u'Unhook the Stars (1996)',
 147: u'Long Kiss Goodnight, The (1996)',
 148: u'Ghost and the Darkness, The (1996)',
 149: u'Jude (1996)',
 150: u'Swingers (1996)',
 151: u'Willy Wonka and the Chocolate Factory (1971)',
 152: u'Sleeper (1973)',
 153: u'Fish Called Wanda, A (1988)',
 154: u"Monty Python's Life of Brian (1979)",
 155: u'Dirty Dancing (1987)',
 156: u'Reservoir Dogs (1992)',
 157: u'Platoon (1986)',
 158: u"Weekend at Bernie's (1989)",
 159: u'Basic Instinct (1992)',
 160: u'Glengarry Glen Ross (1992)',
 161: u'Top Gun (1986)',
 162: u'On Golden Pond (1981)',
 163: u'Return of the Pink Panther, The (1974)',
 164: u'Abyss, The (1989)',
 165: u'Jean de Florette (1986)',
 166: u'Manon of the Spring (Manon des sources) (1986)',
 167: u'Private Benjamin (1980)',
 168: u'Monty Python and the Holy Grail (1974)',
 169: u'Wrong Trousers, The (1993)',
 170: u'Cinema Paradiso (1988)',
 171: u'Delicatessen (1991)',
 172: u'Empire Strikes Back, The (1980)',
 173: u'Princess Bride, The (1987)',
 174: u'Raiders of the Lost Ark (1981)',
 175: u'Brazil (1985)',
 176: u'Aliens (1986)',
 177: u'Good, The Bad and The Ugly, The (1966)',
 178: u'12 Angry Men (1957)',
 179: u'Clockwork Orange, A (1971)',
 180: u'Apocalypse Now (1979)',
 181: u'Return of the Jedi (1983)',
 182: u'GoodFellas (1990)',
 183: u'Alien (1979)',
 184: u'Army of Darkness (1993)',
 185: u'Psycho (1960)',
 186: u'Blues Brothers, The (1980)',
 187: u'Godfather: Part II, The (1974)',
 188: u'Full Metal Jacket (1987)',
 189: u'Grand Day Out, A (1992)',
 190: u'Henry V (1989)',
 191: u'Amadeus (1984)',
 192: u'Raging Bull (1980)',
 193: u'Right Stuff, The (1983)',
 194: u'Sting, The (1973)',
 195: u'Terminator, The (1984)',
 196: u'Dead Poets Society (1989)',
 197: u'Graduate, The (1967)',
 198: u'Nikita (La Femme Nikita) (1990)',
 199: u'Bridge on the River Kwai, The (1957)',
 200: u'Shining, The (1980)',
 201: u'Evil Dead II (1987)',
 202: u'Groundhog Day (1993)',
 203: u'Unforgiven (1992)',
 204: u'Back to the Future (1985)',
 205: u'Patton (1970)',
 206: u'Akira (1988)',
 207: u'Cyrano de Bergerac (1990)',
 208: u'Young Frankenstein (1974)',
 209: u'This Is Spinal Tap (1984)',
 210: u'Indiana Jones and the Last Crusade (1989)',
 211: u'M*A*S*H (1970)',
 212: u'Unbearable Lightness of Being, The (1988)',
 213: u'Room with a View, A (1986)',
 214: u'Pink Floyd - The Wall (1982)',
 215: u'Field of Dreams (1989)',
 216: u'When Harry Met Sally... (1989)',
 217: u"Bram Stoker's Dracula (1992)",
 218: u'Cape Fear (1991)',
 219: u'Nightmare on Elm Street, A (1984)',
 220: u'Mirror Has Two Faces, The (1996)',
 221: u'Breaking the Waves (1996)',
 222: u'Star Trek: First Contact (1996)',
 223: u'Sling Blade (1996)',
 224: u'Ridicule (1996)',
 225: u'101 Dalmatians (1996)',
 226: u'Die Hard 2 (1990)',
 227: u'Star Trek VI: The Undiscovered Country (1991)',
 228: u'Star Trek: The Wrath of Khan (1982)',
 229: u'Star Trek III: The Search for Spock (1984)',
 230: u'Star Trek IV: The Voyage Home (1986)',
 231: u'Batman Returns (1992)',
 232: u'Young Guns (1988)',
 233: u'Under Siege (1992)',
 234: u'Jaws (1975)',
 235: u'Mars Attacks! (1996)',
 236: u'Citizen Ruth (1996)',
 237: u'Jerry Maguire (1996)',
 238: u'Raising Arizona (1987)',
 239: u'Sneakers (1992)',
 240: u'Beavis and Butt-head Do America (1996)',
 241: u'Last of the Mohicans, The (1992)',
 242: u'Kolya (1996)',
 243: u'Jungle2Jungle (1997)',
 244: u"Smilla's Sense of Snow (1997)",
 245: u"Devil's Own, The (1997)",
 246: u'Chasing Amy (1997)',
 247: u'Turbo: A Power Rangers Movie (1997)',
 248: u'Grosse Pointe Blank (1997)',
 249: u'Austin Powers: International Man of Mystery (1997)',
 250: u'Fifth Element, The (1997)',
 251: u'Shall We Dance? (1996)',
 252: u'Lost World: Jurassic Park, The (1997)',
 253: u'Pillow Book, The (1995)',
 254: u'Batman & Robin (1997)',
 255: u"My Best Friend's Wedding (1997)",
 256: u'When the Cats Away (Chacun cherche son chat) (1996)',
 257: u'Men in Black (1997)',
 258: u'Contact (1997)',
 259: u'George of the Jungle (1997)',
 260: u'Event Horizon (1997)',
 261: u'Air Bud (1997)',
 262: u'In the Company of Men (1997)',
 263: u'Steel (1997)',
 264: u'Mimic (1997)',
 265: u'Hunt for Red October, The (1990)',
 266: u'Kull the Conqueror (1997)',
 267: u'unknown',
 268: u'Chasing Amy (1997)',
 269: u'Full Monty, The (1997)',
 270: u'Gattaca (1997)',
 271: u'Starship Troopers (1997)',
 272: u'Good Will Hunting (1997)',
 273: u'Heat (1995)',
 274: u'Sabrina (1995)',
 275: u'Sense and Sensibility (1995)',
 276: u'Leaving Las Vegas (1995)',
 277: u'Restoration (1995)',
 278: u'Bed of Roses (1996)',
 279: u'Once Upon a Time... When We Were Colored (1995)',
 280: u'Up Close and Personal (1996)',
 281: u'River Wild, The (1994)',
 282: u'Time to Kill, A (1996)',
 283: u'Emma (1996)',
 284: u'Tin Cup (1996)',
 285: u'Secrets & Lies (1996)',
 286: u'English Patient, The (1996)',
 287: u"Marvin's Room (1996)",
 288: u'Scream (1996)',
 289: u'Evita (1996)',
 290: u'Fierce Creatures (1997)',
 291: u'Absolute Power (1997)',
 292: u'Rosewood (1997)',
 293: u'Donnie Brasco (1997)',
 294: u'Liar Liar (1997)',
 295: u'Breakdown (1997)',
 296: u'Promesse, La (1996)',
 297: u"Ulee's Gold (1997)",
 298: u'Face/Off (1997)',
 299: u'Hoodlum (1997)',
 300: u'Air Force One (1997)',
 301: u'In & Out (1997)',
 302: u'L.A. Confidential (1997)',
 303: u"Ulee's Gold (1997)",
 304: u'Fly Away Home (1996)',
 305: u'Ice Storm, The (1997)',
 306: u'Mrs. Brown (Her Majesty, Mrs. Brown) (1997)',
 307: u"Devil's Advocate, The (1997)",
 308: u'FairyTale: A True Story (1997)',
 309: u'Deceiver (1997)',
 310: u'Rainmaker, The (1997)',
 311: u'Wings of the Dove, The (1997)',
 312: u'Midnight in the Garden of Good and Evil (1997)',
 313: u'Titanic (1997)',
 314: u'3 Ninjas: High Noon At Mega Mountain (1998)',
 315: u'Apt Pupil (1998)',
 316: u'As Good As It Gets (1997)',
 317: u'In the Name of the Father (1993)',
 318: u"Schindler's List (1993)",
 319: u'Everyone Says I Love You (1996)',
 320: u'Paradise Lost: The Child Murders at Robin Hood Hills (1996)',
 321: u'Mother (1996)',
 322: u'Murder at 1600 (1997)',
 323: u"Dante's Peak (1997)",
 324: u'Lost Highway (1997)',
 325: u'Crash (1996)',
 326: u'G.I. Jane (1997)',
 327: u'Cop Land (1997)',
 328: u'Conspiracy Theory (1997)',
 329: u'Desperate Measures (1998)',
 330: u'187 (1997)',
 331: u'Edge, The (1997)',
 332: u'Kiss the Girls (1997)',
 333: u'Game, The (1997)',
 334: u'U Turn (1997)',
 335: u'How to Be a Player (1997)',
 336: u'Playing God (1997)',
 337: u'House of Yes, The (1997)',
 338: u'Bean (1997)',
 339: u'Mad City (1997)',
 340: u'Boogie Nights (1997)',
 341: u'Critical Care (1997)',
 342: u'Man Who Knew Too Little, The (1997)',
 343: u'Alien: Resurrection (1997)',
 344: u'Apostle, The (1997)',
 345: u'Deconstructing Harry (1997)',
 346: u'Jackie Brown (1997)',
 347: u'Wag the Dog (1997)',
 348: u'Desperate Measures (1998)',
 349: u'Hard Rain (1998)',
 350: u'Fallen (1998)',
 351: u'Prophecy II, The (1998)',
 352: u'Spice World (1997)',
 353: u'Deep Rising (1998)',
 354: u'Wedding Singer, The (1998)',
 355: u'Sphere (1998)',
 356: u'Client, The (1994)',
 357: u"One Flew Over the Cuckoo's Nest (1975)",
 358: u'Spawn (1997)',
 359: u'Assignment, The (1997)',
 360: u'Wonderland (1997)',
 361: u'Incognito (1997)',
 362: u'Blues Brothers 2000 (1998)',
 363: u'Sudden Death (1995)',
 364: u'Ace Ventura: When Nature Calls (1995)',
 365: u'Powder (1995)',
 366: u'Dangerous Minds (1995)',
 367: u'Clueless (1995)',
 368: u'Bio-Dome (1996)',
 369: u'Black Sheep (1996)',
 370: u'Mary Reilly (1996)',
 371: u'Bridges of Madison County, The (1995)',
 372: u'Jeffrey (1995)',
 373: u'Judge Dredd (1995)',
 374: u'Mighty Morphin Power Rangers: The Movie (1995)',
 375: u'Showgirls (1995)',
 376: u'Houseguest (1994)',
 377: u'Heavyweights (1994)',
 378: u'Miracle on 34th Street (1994)',
 379: u'Tales From the Crypt Presents: Demon Knight (1995)',
 380: u'Star Trek: Generations (1994)',
 381: u"Muriel's Wedding (1994)",
 382: u'Adventures of Priscilla, Queen of the Desert, The (1994)',
 383: u'Flintstones, The (1994)',
 384: u'Naked Gun 33 1/3: The Final Insult (1994)',
 385: u'True Lies (1994)',
 386: u'Addams Family Values (1993)',
 387: u'Age of Innocence, The (1993)',
 388: u'Beverly Hills Cop III (1994)',
 389: u'Black Beauty (1994)',
 390: u'Fear of a Black Hat (1993)',
 391: u'Last Action Hero (1993)',
 392: u'Man Without a Face, The (1993)',
 393: u'Mrs. Doubtfire (1993)',
 394: u'Radioland Murders (1994)',
 395: u'Robin Hood: Men in Tights (1993)',
 396: u'Serial Mom (1994)',
 397: u'Striking Distance (1993)',
 398: u'Super Mario Bros. (1993)',
 399: u'Three Musketeers, The (1993)',
 400: u'Little Rascals, The (1994)',
 401: u'Brady Bunch Movie, The (1995)',
 402: u'Ghost (1990)',
 403: u'Batman (1989)',
 404: u'Pinocchio (1940)',
 405: u'Mission: Impossible (1996)',
 406: u'Thinner (1996)',
 407: u'Spy Hard (1996)',
 408: u'Close Shave, A (1995)',
 409: u'Jack (1996)',
 410: u'Kingpin (1996)',
 411: u'Nutty Professor, The (1996)',
 412: u'Very Brady Sequel, A (1996)',
 413: u'Tales from the Crypt Presents: Bordello of Blood (1996)',
 414: u'My Favorite Year (1982)',
 415: u'Apple Dumpling Gang, The (1975)',
 416: u'Old Yeller (1957)',
 417: u'Parent Trap, The (1961)',
 418: u'Cinderella (1950)',
 419: u'Mary Poppins (1964)',
 420: u'Alice in Wonderland (1951)',
 421: u"William Shakespeare's Romeo and Juliet (1996)",
 422: u'Aladdin and the King of Thieves (1996)',
 423: u'E.T. the Extra-Terrestrial (1982)',
 424: u'Children of the Corn: The Gathering (1996)',
 425: u'Bob Roberts (1992)',
 426: u'Transformers: The Movie, The (1986)',
 427: u'To Kill a Mockingbird (1962)',
 428: u'Harold and Maude (1971)',
 429: u'Day the Earth Stood Still, The (1951)',
 430: u'Duck Soup (1933)',
 431: u'Highlander (1986)',
 432: u'Fantasia (1940)',
 433: u'Heathers (1989)',
 434: u'Forbidden Planet (1956)',
 435: u'Butch Cassidy and the Sundance Kid (1969)',
 436: u'American Werewolf in London, An (1981)',
 437: u"Amityville 1992: It's About Time (1992)",
 438: u'Amityville 3-D (1983)',
 439: u'Amityville: A New Generation (1993)',
 440: u'Amityville II: The Possession (1982)',
 441: u'Amityville Horror, The (1979)',
 442: u'Amityville Curse, The (1990)',
 443: u'Birds, The (1963)',
 444: u'Blob, The (1958)',
 445: u'Body Snatcher, The (1945)',
 446: u'Burnt Offerings (1976)',
 447: u'Carrie (1976)',
 448: u'Omen, The (1976)',
 449: u'Star Trek: The Motion Picture (1979)',
 450: u'Star Trek V: The Final Frontier (1989)',
 451: u'Grease (1978)',
 452: u'Jaws 2 (1978)',
 453: u'Jaws 3-D (1983)',
 454: u'Bastard Out of Carolina (1996)',
 455: u"Jackie Chan's First Strike (1996)",
 456: u'Beverly Hills Ninja (1997)',
 457: u'Free Willy 3: The Rescue (1997)',
 458: u'Nixon (1995)',
 459: u'Cry, the Beloved Country (1995)',
 460: u'Crossing Guard, The (1995)',
 461: u'Smoke (1995)',
 462: u'Like Water For Chocolate (Como agua para chocolate) (1992)',
 463: u'Secret of Roan Inish, The (1994)',
 464: u'Vanya on 42nd Street (1994)',
 465: u'Jungle Book, The (1994)',
 466: u'Red Rock West (1992)',
 467: u'Bronx Tale, A (1993)',
 468: u'Rudy (1993)',
 469: u'Short Cuts (1993)',
 470: u'Tombstone (1993)',
 471: u'Courage Under Fire (1996)',
 472: u'Dragonheart (1996)',
 473: u'James and the Giant Peach (1996)',
 474: u'Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1963)',
 475: u'Trainspotting (1996)',
 476: u'First Wives Club, The (1996)',
 477: u'Matilda (1996)',
 478: u'Philadelphia Story, The (1940)',
 479: u'Vertigo (1958)',
 480: u'North by Northwest (1959)',
 481: u'Apartment, The (1960)',
 482: u'Some Like It Hot (1959)',
 483: u'Casablanca (1942)',
 484: u'Maltese Falcon, The (1941)',
 485: u'My Fair Lady (1964)',
 486: u'Sabrina (1954)',
 487: u'Roman Holiday (1953)',
 488: u'Sunset Blvd. (1950)',
 489: u'Notorious (1946)',
 490: u'To Catch a Thief (1955)',
 491: u'Adventures of Robin Hood, The (1938)',
 492: u'East of Eden (1955)',
 493: u'Thin Man, The (1934)',
 494: u'His Girl Friday (1940)',
 495: u'Around the World in 80 Days (1956)',
 496: u"It's a Wonderful Life (1946)",
 497: u'Bringing Up Baby (1938)',
 498: u'African Queen, The (1951)',
 499: u'Cat on a Hot Tin Roof (1958)',
 500: u'Fly Away Home (1996)',
 501: u'Dumbo (1941)',
 502: u'Bananas (1971)',
 503: u'Candidate, The (1972)',
 504: u'Bonnie and Clyde (1967)',
 505: u'Dial M for Murder (1954)',
 506: u'Rebel Without a Cause (1955)',
 507: u'Streetcar Named Desire, A (1951)',
 508: u'People vs. Larry Flynt, The (1996)',
 509: u'My Left Foot (1989)',
 510: u'Magnificent Seven, The (1954)',
 511: u'Lawrence of Arabia (1962)',
 512: u'Wings of Desire (1987)',
 513: u'Third Man, The (1949)',
 514: u'Annie Hall (1977)',
 515: u'Boot, Das (1981)',
 516: u'Local Hero (1983)',
 517: u'Manhattan (1979)',
 518: u"Miller's Crossing (1990)",
 519: u'Treasure of the Sierra Madre, The (1948)',
 520: u'Great Escape, The (1963)',
 521: u'Deer Hunter, The (1978)',
 522: u'Down by Law (1986)',
 523: u'Cool Hand Luke (1967)',
 524: u'Great Dictator, The (1940)',
 525: u'Big Sleep, The (1946)',
 526: u'Ben-Hur (1959)',
 527: u'Gandhi (1982)',
 528: u'Killing Fields, The (1984)',
 529: u'My Life as a Dog (Mitt liv som hund) (1985)',
 530: u'Man Who Would Be King, The (1975)',
 531: u'Shine (1996)',
 532: u'Kama Sutra: A Tale of Love (1996)',
 533: u'Daytrippers, The (1996)',
 534: u'Traveller (1997)',
 535: u'Addicted to Love (1997)',
 536: u'Ponette (1996)',
 537: u'My Own Private Idaho (1991)',
 538: u'Anastasia (1997)',
 539: u'Mouse Hunt (1997)',
 540: u'Money Train (1995)',
 541: u'Mortal Kombat (1995)',
 542: u'Pocahontas (1995)',
 543: u'Mis\ufffdrables, Les (1995)',
 544: u"Things to Do in Denver when You're Dead (1995)",
 545: u'Vampire in Brooklyn (1995)',
 546: u'Broken Arrow (1996)',
 547: u"Young Poisoner's Handbook, The (1995)",
 548: u'NeverEnding Story III, The (1994)',
 549: u'Rob Roy (1995)',
 550: u'Die Hard: With a Vengeance (1995)',
 551: u'Lord of Illusions (1995)',
 552: u'Species (1995)',
 553: u'Walk in the Clouds, A (1995)',
 554: u'Waterworld (1995)',
 555: u"White Man's Burden (1995)",
 556: u'Wild Bill (1995)',
 557: u'Farinelli: il castrato (1994)',
 558: u'Heavenly Creatures (1994)',
 559: u'Interview with the Vampire (1994)',
 560: u"Kid in King Arthur's Court, A (1995)",
 561: u"Mary Shelley's Frankenstein (1994)",
 562: u'Quick and the Dead, The (1995)',
 563: u"Stephen King's The Langoliers (1995)",
 564: u'Tales from the Hood (1995)',
 565: u'Village of the Damned (1995)',
 566: u'Clear and Present Danger (1994)',
 567: u"Wes Craven's New Nightmare (1994)",
 568: u'Speed (1994)',
 569: u'Wolf (1994)',
 570: u'Wyatt Earp (1994)',
 571: u'Another Stakeout (1993)',
 572: u'Blown Away (1994)',
 573: u'Body Snatchers (1993)',
 574: u'Boxing Helena (1993)',
 575: u"City Slickers II: The Legend of Curly's Gold (1994)",
 576: u'Cliffhanger (1993)',
 577: u'Coneheads (1993)',
 578: u'Demolition Man (1993)',
 579: u'Fatal Instinct (1993)',
 580: u'Englishman Who Went Up a Hill, But Came Down a Mountain, The (1995)',
 581: u'Kalifornia (1993)',
 582: u'Piano, The (1993)',
 583: u'Romeo Is Bleeding (1993)',
 584: u'Secret Garden, The (1993)',
 585: u'Son in Law (1993)',
 586: u'Terminal Velocity (1994)',
 587: u'Hour of the Pig, The (1993)',
 588: u'Beauty and the Beast (1991)',
 589: u'Wild Bunch, The (1969)',
 590: u'Hellraiser: Bloodline (1996)',
 591: u'Primal Fear (1996)',
 592: u'True Crime (1995)',
 593: u'Stalingrad (1993)',
 594: u'Heavy (1995)',
 595: u'Fan, The (1996)',
 596: u'Hunchback of Notre Dame, The (1996)',
 597: u'Eraser (1996)',
 598: u'Big Squeeze, The (1996)',
 599: u'Police Story 4: Project S (Chao ji ji hua) (1993)',
 600: u"Daniel Defoe's Robinson Crusoe (1996)",
 601: u'For Whom the Bell Tolls (1943)',
 602: u'American in Paris, An (1951)',
 603: u'Rear Window (1954)',
 604: u'It Happened One Night (1934)',
 605: u'Meet Me in St. Louis (1944)',
 606: u'All About Eve (1950)',
 607: u'Rebecca (1940)',
 608: u'Spellbound (1945)',
 609: u'Father of the Bride (1950)',
 610: u'Gigi (1958)',
 611: u'Laura (1944)',
 612: u'Lost Horizon (1937)',
 613: u'My Man Godfrey (1936)',
 614: u'Giant (1956)',
 615: u'39 Steps, The (1935)',
 616: u'Night of the Living Dead (1968)',
 617: u'Blue Angel, The (Blaue Engel, Der) (1930)',
 618: u'Picnic (1955)',
 619: u'Extreme Measures (1996)',
 620: u'Chamber, The (1996)',
 621: u'Davy Crockett, King of the Wild Frontier (1955)',
 622: u'Swiss Family Robinson (1960)',
 623: u'Angels in the Outfield (1994)',
 624: u'Three Caballeros, The (1945)',
 625: u'Sword in the Stone, The (1963)',
 626: u'So Dear to My Heart (1949)',
 627: u'Robin Hood: Prince of Thieves (1991)',
 628: u'Sleepers (1996)',
 629: u'Victor/Victoria (1982)',
 630: u'Great Race, The (1965)',
 631: u'Crying Game, The (1992)',
 632: u"Sophie's Choice (1982)",
 633: u'Christmas Carol, A (1938)',
 634: u"Microcosmos: Le peuple de l'herbe (1996)",
 635: u'Fog, The (1980)',
 636: u'Escape from New York (1981)',
 637: u'Howling, The (1981)',
 638: u'Return of Martin Guerre, The (Retour de Martin Guerre, Le) (1982)',
 639: u'Tin Drum, The (Blechtrommel, Die) (1979)',
 640: u'Cook the Thief His Wife & Her Lover, The (1989)',
 641: u'Paths of Glory (1957)',
 642: u'Grifters, The (1990)',
 643: u'The Innocent (1994)',
 644: u'Thin Blue Line, The (1988)',
 645: u'Paris Is Burning (1990)',
 646: u'Once Upon a Time in the West (1969)',
 647: u'Ran (1985)',
 648: u'Quiet Man, The (1952)',
 649: u'Once Upon a Time in America (1984)',
 650: u'Seventh Seal, The (Sjunde inseglet, Det) (1957)',
 651: u'Glory (1989)',
 652: u'Rosencrantz and Guildenstern Are Dead (1990)',
 653: u'Touch of Evil (1958)',
 654: u'Chinatown (1974)',
 655: u'Stand by Me (1986)',
 656: u'M (1931)',
 657: u'Manchurian Candidate, The (1962)',
 658: u'Pump Up the Volume (1990)',
 659: u'Arsenic and Old Lace (1944)',
 660: u'Fried Green Tomatoes (1991)',
 661: u'High Noon (1952)',
 662: u'Somewhere in Time (1980)',
 663: u'Being There (1979)',
 664: u'Paris, Texas (1984)',
 665: u'Alien 3 (1992)',
 666: u"Blood For Dracula (Andy Warhol's Dracula) (1974)",
 667: u'Audrey Rose (1977)',
 668: u'Blood Beach (1981)',
 669: u'Body Parts (1991)',
 670: u'Body Snatchers (1993)',
 671: u'Bride of Frankenstein (1935)',
 672: u'Candyman (1992)',
 673: u'Cape Fear (1962)',
 674: u'Cat People (1982)',
 675: u'Nosferatu (Nosferatu, eine Symphonie des Grauens) (1922)',
 676: u'Crucible, The (1996)',
 677: u'Fire on the Mountain (1996)',
 678: u'Volcano (1997)',
 679: u'Conan the Barbarian (1981)',
 680: u'Kull the Conqueror (1997)',
 681: u'Wishmaster (1997)',
 682: u'I Know What You Did Last Summer (1997)',
 683: u'Rocket Man (1997)',
 684: u'In the Line of Fire (1993)',
 685: u'Executive Decision (1996)',
 686: u'Perfect World, A (1993)',
 687: u"McHale's Navy (1997)",
 688: u'Leave It to Beaver (1997)',
 689: u'Jackal, The (1997)',
 690: u'Seven Years in Tibet (1997)',
 691: u'Dark City (1998)',
 692: u'American President, The (1995)',
 693: u'Casino (1995)',
 694: u'Persuasion (1995)',
 695: u'Kicking and Screaming (1995)',
 696: u'City Hall (1996)',
 697: u'Basketball Diaries, The (1995)',
 698: u'Browning Version, The (1994)',
 699: u'Little Women (1994)',
 700: u'Miami Rhapsody (1995)',
 701: u'Wonderful, Horrible Life of Leni Riefenstahl, The (1993)',
 702: u'Barcelona (1994)',
 703: u"Widows' Peak (1994)",
 704: u'House of the Spirits, The (1993)',
 705: u"Singin' in the Rain (1952)",
 706: u'Bad Moon (1996)',
 707: u'Enchanted April (1991)',
 708: u'Sex, Lies, and Videotape (1989)',
 709: u'Strictly Ballroom (1992)',
 710: u'Better Off Dead... (1985)',
 711: u'Substance of Fire, The (1996)',
 712: u'Tin Men (1987)',
 713: u'Othello (1995)',
 714: u'Carrington (1995)',
 715: u'To Die For (1995)',
 716: u'Home for the Holidays (1995)',
 717: u'Juror, The (1996)',
 718: u'In the Bleak Midwinter (1995)',
 719: u'Canadian Bacon (1994)',
 720: u'First Knight (1995)',
 721: u'Mallrats (1995)',
 722: u'Nine Months (1995)',
 723: u'Boys on the Side (1995)',
 724: u'Circle of Friends (1995)',
 725: u'Exit to Eden (1994)',
 726: u'Fluke (1995)',
 727: u'Immortal Beloved (1994)',
 728: u'Junior (1994)',
 729: u'Nell (1994)',
 730: u'Queen Margot (Reine Margot, La) (1994)',
 731: u'Corrina, Corrina (1994)',
 732: u'Dave (1993)',
 733: u'Go Fish (1994)',
 734: u'Made in America (1993)',
 735: u'Philadelphia (1993)',
 736: u'Shadowlands (1993)',
 737: u'Sirens (1994)',
 738: u'Threesome (1994)',
 739: u'Pretty Woman (1990)',
 740: u'Jane Eyre (1996)',
 741: u'Last Supper, The (1995)',
 742: u'Ransom (1996)',
 743: u'Crow: City of Angels, The (1996)',
 744: u'Michael Collins (1996)',
 745: u'Ruling Class, The (1972)',
 746: u'Real Genius (1985)',
 747: u'Benny & Joon (1993)',
 748: u'Saint, The (1997)',
 749: u'MatchMaker, The (1997)',
 750: u'Amistad (1997)',
 751: u'Tomorrow Never Dies (1997)',
 752: u'Replacement Killers, The (1998)',
 753: u'Burnt By the Sun (1994)',
 754: u'Red Corner (1997)',
 755: u'Jumanji (1995)',
 756: u'Father of the Bride Part II (1995)',
 757: u'Across the Sea of Time (1995)',
 758: u'Lawnmower Man 2: Beyond Cyberspace (1996)',
 759: u'Fair Game (1995)',
 760: u'Screamers (1995)',
 761: u'Nick of Time (1995)',
 762: u'Beautiful Girls (1996)',
 763: u'Happy Gilmore (1996)',
 764: u'If Lucy Fell (1996)',
 765: u'Boomerang (1992)',
 766: u'Man of the Year (1995)',
 767: u'Addiction, The (1995)',
 768: u'Casper (1995)',
 769: u'Congo (1995)',
 770: u'Devil in a Blue Dress (1995)',
 771: u'Johnny Mnemonic (1995)',
 772: u'Kids (1995)',
 773: u'Mute Witness (1994)',
 774: u'Prophecy, The (1995)',
 775: u'Something to Talk About (1995)',
 776: u'Three Wishes (1995)',
 777: u'Castle Freak (1995)',
 778: u'Don Juan DeMarco (1995)',
 779: u'Drop Zone (1994)',
 780: u'Dumb & Dumber (1994)',
 781: u'French Kiss (1995)',
 782: u'Little Odessa (1994)',
 783: u'Milk Money (1994)',
 784: u'Beyond Bedlam (1993)',
 785: u'Only You (1994)',
 786: u'Perez Family, The (1995)',
 787: u'Roommates (1995)',
 788: u'Relative Fear (1994)',
 789: u'Swimming with Sharks (1995)',
 790: u'Tommy Boy (1995)',
 791: u'Baby-Sitters Club, The (1995)',
 792: u'Bullets Over Broadway (1994)',
 793: u'Crooklyn (1994)',
 794: u'It Could Happen to You (1994)',
 795: u'Richie Rich (1994)',
 796: u'Speechless (1994)',
 797: u'Timecop (1994)',
 798: u'Bad Company (1995)',
 799: u'Boys Life (1995)',
 800: u'In the Mouth of Madness (1995)',
 801: u'Air Up There, The (1994)',
 802: u'Hard Target (1993)',
 803: u'Heaven & Earth (1993)',
 804: u'Jimmy Hollywood (1994)',
 805: u'Manhattan Murder Mystery (1993)',
 806: u'Menace II Society (1993)',
 807: u'Poetic Justice (1993)',
 808: u'Program, The (1993)',
 809: u'Rising Sun (1993)',
 810: u'Shadow, The (1994)',
 811: u'Thirty-Two Short Films About Glenn Gould (1993)',
 812: u'Andre (1994)',
 813: u'Celluloid Closet, The (1995)',
 814: u'Great Day in Harlem, A (1994)',
 815: u'One Fine Day (1996)',
 816: u'Candyman: Farewell to the Flesh (1995)',
 817: u'Frisk (1995)',
 818: u'Girl 6 (1996)',
 819: u'Eddie (1996)',
 820: u'Space Jam (1996)',
 821: u'Mrs. Winterbourne (1996)',
 822: u'Faces (1968)',
 823: u'Mulholland Falls (1996)',
 824: u'Great White Hype, The (1996)',
 825: u'Arrival, The (1996)',
 826: u'Phantom, The (1996)',
 827: u'Daylight (1996)',
 828: u'Alaska (1996)',
 829: u'Fled (1996)',
 830: u'Power 98 (1995)',
 831: u'Escape from L.A. (1996)',
 832: u'Bogus (1996)',
 833: u'Bulletproof (1996)',
 834: u'Halloween: The Curse of Michael Myers (1995)',
 835: u'Gay Divorcee, The (1934)',
 836: u'Ninotchka (1939)',
 837: u'Meet John Doe (1941)',
 838: u'In the Line of Duty 2 (1987)',
 839: u'Loch Ness (1995)',
 840: u'Last Man Standing (1996)',
 841: u'Glimmer Man, The (1996)',
 842: u'Pollyanna (1960)',
 843: u'Shaggy Dog, The (1959)',
 844: u'Freeway (1996)',
 845: u'That Thing You Do! (1996)',
 846: u'To Gillian on Her 37th Birthday (1996)',
 847: u'Looking for Richard (1996)',
 848: u'Murder, My Sweet (1944)',
 849: u'Days of Thunder (1990)',
 850: u'Perfect Candidate, A (1996)',
 851: u'Two or Three Things I Know About Her (1966)',
 852: u'Bloody Child, The (1996)',
 853: u'Braindead (1992)',
 854: u'Bad Taste (1987)',
 855: u'Diva (1981)',
 856: u'Night on Earth (1991)',
 857: u'Paris Was a Woman (1995)',
 858: u'Amityville: Dollhouse (1996)',
 859: u"April Fool's Day (1986)",
 860: u'Believers, The (1987)',
 861: u'Nosferatu a Venezia (1986)',
 862: u'Jingle All the Way (1996)',
 863: u'Garden of Finzi-Contini, The (Giardino dei Finzi-Contini, Il) (1970)',
 864: u'My Fellow Americans (1996)',
 865: u'Ice Storm, The (1997)',
 866: u'Michael (1996)',
 867: u'Whole Wide World, The (1996)',
 868: u'Hearts and Minds (1996)',
 869: u'Fools Rush In (1997)',
 870: u'Touch (1997)',
 871: u'Vegas Vacation (1997)',
 872: u'Love Jones (1997)',
 873: u'Picture Perfect (1997)',
 874: u'Career Girls (1997)',
 875: u"She's So Lovely (1997)",
 876: u'Money Talks (1997)',
 877: u'Excess Baggage (1997)',
 878: u'That Darn Cat! (1997)',
 879: u'Peacemaker, The (1997)',
 880: u'Soul Food (1997)',
 881: u'Money Talks (1997)',
 882: u'Washington Square (1997)',
 883: u'Telling Lies in America (1997)',
 884: u'Year of the Horse (1997)',
 885: u'Phantoms (1998)',
 886: u'Life Less Ordinary, A (1997)',
 887: u"Eve's Bayou (1997)",
 888: u'One Night Stand (1997)',
 889: u'Tango Lesson, The (1997)',
 890: u'Mortal Kombat: Annihilation (1997)',
 891: u'Bent (1997)',
 892: u'Flubber (1997)',
 893: u'For Richer or Poorer (1997)',
 894: u'Home Alone 3 (1997)',
 895: u'Scream 2 (1997)',
 896: u'Sweet Hereafter, The (1997)',
 897: u'Time Tracers (1995)',
 898: u'Postman, The (1997)',
 899: u'Winter Guest, The (1997)',
 900: u'Kundun (1997)',
 901: u'Mr. Magoo (1997)',
 902: u'Big Lebowski, The (1998)',
 903: u'Afterglow (1997)',
 904: u'Ma vie en rose (My Life in Pink) (1997)',
 905: u'Great Expectations (1998)',
 906: u'Oscar & Lucinda (1997)',
 907: u'Vermin (1998)',
 908: u'Half Baked (1998)',
 909: u'Dangerous Beauty (1998)',
 910: u'Nil By Mouth (1997)',
 911: u'Twilight (1998)',
 912: u'U.S. Marshalls (1998)',
 913: u'Love and Death on Long Island (1997)',
 914: u'Wild Things (1998)',
 915: u'Primary Colors (1998)',
 916: u'Lost in Space (1998)',
 917: u'Mercury Rising (1998)',
 918: u'City of Angels (1998)',
 919: u'City of Lost Children, The (1995)',
 920: u'Two Bits (1995)',
 921: u'Farewell My Concubine (1993)',
 922: u'Dead Man (1995)',
 923: u'Raise the Red Lantern (1991)',
 924: u'White Squall (1996)',
 925: u'Unforgettable (1996)',
 926: u'Down Periscope (1996)',
 927: u'Flower of My Secret, The (Flor de mi secreto, La) (1995)',
 928: u'Craft, The (1996)',
 929: u'Harriet the Spy (1996)',
 930: u'Chain Reaction (1996)',
 931: u'Island of Dr. Moreau, The (1996)',
 932: u'First Kid (1996)',
 933: u'Funeral, The (1996)',
 934: u"Preacher's Wife, The (1996)",
 935: u'Paradise Road (1997)',
 936: u'Brassed Off (1996)',
 937: u'Thousand Acres, A (1997)',
 938: u'Smile Like Yours, A (1997)',
 939: u'Murder in the First (1995)',
 940: u'Airheads (1994)',
 941: u'With Honors (1994)',
 942: u"What's Love Got to Do with It (1993)",
 943: u'Killing Zoe (1994)',
 944: u'Renaissance Man (1994)',
 945: u'Charade (1963)',
 946: u'Fox and the Hound, The (1981)',
 947: u'Big Blue, The (Grand bleu, Le) (1988)',
 948: u'Booty Call (1997)',
 949: u'How to Make an American Quilt (1995)',
 950: u'Georgia (1995)',
 951: u'Indian in the Cupboard, The (1995)',
 952: u'Blue in the Face (1995)',
 953: u'Unstrung Heroes (1995)',
 954: u'Unzipped (1995)',
 955: u'Before Sunrise (1995)',
 956: u"Nobody's Fool (1994)",
 957: u'Pushing Hands (1992)',
 958: u'To Live (Huozhe) (1994)',
 959: u'Dazed and Confused (1993)',
 960: u'Naked (1993)',
 961: u'Orlando (1993)',
 962: u'Ruby in Paradise (1993)',
 963: u'Some Folks Call It a Sling Blade (1993)',
 964: u'Month by the Lake, A (1995)',
 965: u'Funny Face (1957)',
 966: u'Affair to Remember, An (1957)',
 967: u'Little Lord Fauntleroy (1936)',
 968: u'Inspector General, The (1949)',
 969: u'Winnie the Pooh and the Blustery Day (1968)',
 970: u'Hear My Song (1991)',
 971: u'Mediterraneo (1991)',
 972: u'Passion Fish (1992)',
 973: u'Grateful Dead (1995)',
 974: u'Eye for an Eye (1996)',
 975: u'Fear (1996)',
 976: u'Solo (1996)',
 977: u'Substitute, The (1996)',
 978: u"Heaven's Prisoners (1996)",
 979: u'Trigger Effect, The (1996)',
 980: u'Mother Night (1996)',
 981: u'Dangerous Ground (1997)',
 982: u'Maximum Risk (1996)',
 983: u"Rich Man's Wife, The (1996)",
 984: u'Shadow Conspiracy (1997)',
 985: u'Blood & Wine (1997)',
 986: u'Turbulence (1997)',
 987: u'Underworld (1997)',
 988: u'Beautician and the Beast, The (1997)',
 989: u"Cats Don't Dance (1997)",
 990: u'Anna Karenina (1997)',
 991: u'Keys to Tulsa (1997)',
 992: u'Head Above Water (1996)',
 993: u'Hercules (1997)',
 994: u'Last Time I Committed Suicide, The (1997)',
 995: u'Kiss Me, Guido (1997)',
 996: u'Big Green, The (1995)',
 997: u'Stuart Saves His Family (1995)',
 998: u'Cabin Boy (1994)',
 999: u'Clean Slate (1994)',
 1000: u'Lightning Jack (1994)',
 ...}

r_with_titles = r_123.map(lambda x: (titles[x.product], x.rating))
r_with_titles.sortBy(lambda x:-x[1]).take(10)

[(u'Fantasia (1940)', 5.0),
 (u'Postino, Il (1994)', 5.0),
 (u'2001: A Space Odyssey (1968)', 5.0),
 (u'Raging Bull (1980)', 5.0),
 (u'Jean de Florette (1986)', 5.0),
 (u'Secrets & Lies (1996)', 5.0),
 (u'My Fair Lady (1964)', 5.0),
 (u'Godfather, The (1972)', 5.0),
 (u'Lawrence of Arabia (1962)', 5.0),
 (u'Enchanted April (1991)', 5.0)]

topK_with_titles = map(lambda x:(titles[x.product], x.rating), topK_movies)
topK_with_titles

[(u"Marvin's Room (1996)", 6.238110881748787),
 (u'Full Monty, The (1997)', 6.168910818255823),
 (u'Boot, Das (1981)', 5.954825359796975),
 (u'Old Yeller (1957)', 5.921099652606939),
 (u'Candidate, The (1972)', 5.670748725720875)]

相似度計算

sampleRDD = sc.parallelize([['A', 111], ['A', 222], ['B', 333]])
sampleRDD.lookup('A')

[111, 222]

使用餘弦相似度來計算電影之間的相似程度，比如我們要計算編號爲456的電影與其他電影的餘弦相似度，首先我們要將電影456的因子提取出來：

arr = cf_model.productFeatures().lookup(456)[0]
arr

array('d', [0.029076049104332924, 0.21009714901447296, 0.0290555227547884, -0.5571964383125305, -0.3824714124202728, -0.4592842161655426, 0.6329585313796997, -0.362333744764328, -0.1305536925792694, 0.8419598340988159, -0.13552409410476685, -0.6138198971748352, 0.02604905515909195, 0.08060657978057861, -0.16706441342830658, -0.3220045566558838, 0.43676093220710754, 0.07212082296609879, 0.16547970473766327, 0.049271613359451294, -0.018478330224752426, 0.4917396306991577, -1.259914517402649, 0.30777591466903687, 0.3512609004974365, -1.1641650199890137, -0.08893561363220215, 0.5041327476501465, -0.5516676902770996, -0.13129214942455292, -0.7094163298606873, -0.095136858522892, 0.0024825106374919415, -0.5574610233306885, 0.6876130104064941, -0.14038291573524475, -0.3861311674118042, 0.08736740052700043, 0.7943630218505859, 1.0195096731185913, 0.49429407715797424, -0.07107719779014587, 0.21480131149291992, -0.572085976600647, 0.030756879597902298, 1.120257019996643, 0.012996670790016651, 0.5901889801025391, 0.9270225167274475, -0.8173779845237732])

# 與lookup效果一樣
cf_model.productFeatures().filter(lambda x:x[0] == 456)\
.map(lambda x:x[1]).first()

array('d', [0.029076049104332924, 0.21009714901447296, 0.0290555227547884, -0.5571964383125305, -0.3824714124202728, -0.4592842161655426, 0.6329585313796997, -0.362333744764328, -0.1305536925792694, 0.8419598340988159, -0.13552409410476685, -0.6138198971748352, 0.02604905515909195, 0.08060657978057861, -0.16706441342830658, -0.3220045566558838, 0.43676093220710754, 0.07212082296609879, 0.16547970473766327, 0.049271613359451294, -0.018478330224752426, 0.4917396306991577, -1.259914517402649, 0.30777591466903687, 0.3512609004974365, -1.1641650199890137, -0.08893561363220215, 0.5041327476501465, -0.5516676902770996, -0.13129214942455292, -0.7094163298606873, -0.095136858522892, 0.0024825106374919415, -0.5574610233306885, 0.6876130104064941, -0.14038291573524475, -0.3861311674118042, 0.08736740052700043, 0.7943630218505859, 1.0195096731185913, 0.49429407715797424, -0.07107719779014587, 0.21480131149291992, -0.572085976600647, 0.030756879597902298, 1.120257019996643, 0.012996670790016651, 0.5901889801025391, 0.9270225167274475, -0.8173779845237732])

提取的電影456的因子以數組形式返回，爲了計算餘弦相似度，需要將其向量化

from pyspark.mllib.linalg import DenseVector

selectedVector = DenseVector(arr)
selectedVector

DenseVector([0.0291, 0.2101, 0.0291, -0.5572, -0.3825, -0.4593, 0.633, -0.3623, -0.1306, 0.842, -0.1355, -0.6138, 0.026, 0.0806, -0.1671, -0.322, 0.4368, 0.0721, 0.1655, 0.0493, -0.0185, 0.4917, -1.2599, 0.3078, 0.3513, -1.1642, -0.0889, 0.5041, -0.5517, -0.1313, -0.7094, -0.0951, 0.0025, -0.5575, 0.6876, -0.1404, -0.3861, 0.0874, 0.7944, 1.0195, 0.4943, -0.0711, 0.2148, -0.5721, 0.0308, 1.1203, 0.013, 0.5902, 0.927, -0.8174])

# 定義餘弦相似度函數
def cosSim(vectorA, vectorB):
    return vectorA.dot(vectorB) / (vectorA.norm(2)*vectorB.norm(2))

cosSim(selectedVector, selectedVector)

1.0

使用map方法將cosSim函數映射到每一個電影的因子上，返回由電影編號和餘弦相似度組成的元組

sims = cf_model.productFeatures()\
.map(lambda x:(x[0], cosSim(selectedVector, DenseVector(x[1]))))
sims.take(5)

[(2, 0.63205163077832793),
 (4, 0.57081651505456033),
 (6, 0.57056619721078805),
 (8, 0.61730808637739021),
 (10, 0.55560185135898443)]

取相似度最高的10部電影

simsTopK = sims.top(10, lambda x:x[1])
simsTopK

[(456, 1.0),
 (1446, 0.77381634899106144),
 (249, 0.75186850153352536),
 (1206, 0.75042081098056868),
 (1028, 0.74412474419118724),
 (1435, 0.7440397393142627),
 (42, 0.73813968356865434),
 (1249, 0.73347222767192244),
 (411, 0.73245706263195443),
 (240, 0.7307674356843924)]

使用top方法和takeOrdered方法的效率比較高，因爲只要將指定的記錄返回就可以了，不需要對所有記錄都進行排序；而使用sortBy方法的執行效率則較低，因爲這要將所有記錄都排序之後再選擇記錄。

sims.takeOrdered(10, lambda x:-x[1])
sims.sortBy(lambda x:x[1], False).take(10)

[(456, 1.0),
 (1446, 0.77381634899106144),
 (249, 0.75186850153352536),
 (1206, 0.75042081098056868),
 (1028, 0.74412474419118724),
 (1435, 0.7440397393142627),
 (42, 0.73813968356865434),
 (1249, 0.73347222767192244),
 (411, 0.73245706263195443),
 (240, 0.7307674356843924)]

map( lambda x:(titles[x[0]], x[1]), simsTopK)

[(u'Beverly Hills Ninja (1997)', 1.0),
 (u'Bye Bye, Love (1995)', 0.77381634899106144),
 (u'Austin Powers: International Man of Mystery (1997)', 0.75186850153352536),
 (u'Amos & Andrew (1993)', 0.75042081098056868),
 (u'Grumpier Old Men (1995)', 0.74412474419118724),
 (u'Steal Big, Steal Little (1995)', 0.7440397393142627),
 (u'Clerks (1994)', 0.73813968356865434),
 (u'For Love or Money (1993)', 0.73347222767192244),
 (u'Nutty Professor, The (1996)', 0.73245706263195443),
 (u'Beavis and Butt-head Do America (1996)', 0.7307674356843924)]

模型驗證

MSE\RMSE\MAE

mllib在其evaluation模塊中提供了相應的RegressionMetrics類，可用於計算MSE、 RMSE和MAE，該類只需要傳入一個由“實際評分-預測評分”組成的RDD即可生成相應對象

actual = ratings.map(lambda r: ((r.user, r.product), r.rating))
prediction = cf_model.predictAll(actual.map(lambda x: x[0]))\
                     .map(lambda r: ((r.user, r.product), r.rating))
actual_prediction = actual.join(prediction)
actual_prediction.take(5)

[((506, 568), (5.0, 4.495298253573846)),
 ((109, 365), (4.0, 4.068188145295891)),
 ((621, 577), (3.0, 3.2018734286795425)),
 ((720, 286), (5.0, 4.963056006543837)),
 ((812, 326), (4.0, 3.999871511192283))]

from pyspark.mllib.evaluation import RegressionMetrics

metrics = RegressionMetrics(actual_prediction.map(lambda x: x[1]))

print 'MSE =', metrics.meanSquaredError
print 'RMSE =', metrics.rootMeanSquaredError
print 'MAE =', metrics.meanAbsoluteError

MSE = 0.0845211871908
RMSE = 0.290725277867
MAE = 0.204405585188

MAP

Mllib在其evaluation模塊中有一個RankingMetrics類，可以很方便地計算PK和MAP。該類需要傳入一個“(Prediction, Labels)”類型的RDD，其中Prediction是某個用戶按模型預測排序的產品列表，Labels爲該用戶實際購買的產品列表

productIDs = cf_model.productFeatures().map(lambda p: p[0]).collect()
productMatrix = cf_model.productFeatures().map(lambda p: p[1]).collect()

產品因子矩陣與每一個用戶因子矩陣做點積計算預測評分，把相對應的電影ID關聯進去之後排序，排序後預測評分就不需要，只保留排過序的電影ID

import numpy as np

estRatings = cf_model.userFeatures()\
.map(lambda x: (x[0], list(np.dot(productMatrix, x[1]))))\
.map(lambda x: (x[0], zip(x[1], productIDs)))\
.map(lambda x: (x[0], sorted(x[1] ,key=(lambda m: m[0]), reverse=True)))\
.map(lambda x: (x[0], [i[1] for i in x[1]]))
estRatings.first()

用戶實際評分的電影，要生成的是key-value類型的RDD

userMovies = ratings.map(lambda r:(r.user, r.product))\
                    .groupByKey()\
                    .mapValues(list)
userMovies.first()

userMovies與estRatings具有相似的內部結構，將這兩個RDD連接起來，再提取其中的預測結果和實際結果，就構成了需要傳入RankingMetrics類的RDD

predictionAndLabels = estRatings.join(userMovies).map(lambda x: x[1])
predictionAndLabels.first()

計算PK和MAP

from pyspark.mllib.evaluation import RankingMetrics

rankingMetrics = RankingMetrics(predictionAndLabels)
print 'MAP =', rankingMetrics.meanAveragePrecision
print 'PrecisionAtK =', rankingMetrics.precisionAt(20)

MAP = 0.192062343904
PrecisionAtK = 0.182025450689

模型持久化

cf_model.save(sc, './cf_model')

from pyspark.mllib.recommendation import MatrixFactorizationModel

load_cf_model = MatrixFactorizationModel.load(sc, './cf_model')
load_cf_model.predict(123, 456)

0.5189201089615622

滌生（bluez）

發佈了139 篇原創文章 · 獲贊 32 · 訪問量 2萬+

私信關注

movie_recommendation_spark1

mllib建立推薦模型

數據準備

建模

相似度計算

模型驗證

MSE\RMSE\MAE

MAP

模型持久化

LeetCode------整數拆分

LeetCode------尋找兩個有序數組的中位數

Course_clustering_model

chapter11(sklearn)

第二階段：分佈式理論簡介：2.3 一致性、2PC和3PC

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結