# Movie_Recommend wires a Spark ALS pipeline to an SSM site, and has been superseded

> Movie_Recommend is a MIT licensed graduation project that pairs an SSM web site and an admin back end with a Spark recommendation pipeline over MovieLens 1M, ingesting click events through nginx, Flume and Kafka. Its README now points readers to a separate rewrite, CineMatch, for anything new.

**CodeRayZhang/Movie_Recommend** — 基于Spark的电影推荐系统，包含爬虫项目、web网站、后台管理系统以及spark推荐系统

- Repository: https://github.com/CodeRayZhang/Movie_Recommend
- Stars: 3,013 · Forks: 1,049
- Language: Java
- License: MIT
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/coderayzhang-movie-recommend

## A graduation project, superseded by its own rewrite

The first thing the README now says is that the project has been modernised in a separate repository, CineMatch, described as a movie recommendation system focused on movie discovery, personalised recommendations and personal watchlists, with the new page linked for running or continuing development.

So this repository is now a legacy archive with an exit sign on it. The README is explicit about the status of each piece: it keeps the original graduation project code, architecture diagrams and deployment instructions, and everything below that point describes the old version.

The separation matters more than a version bump. CineMatch uses an independent codebase and database and can run on its own without starting the old Spark or Hadoop components. Old accounts, historical ratings and the old ALS model do not migrate automatically, and the new models have to be retrained and enabled following the training guide.

Two audiences remain, and they are different. One is reading the old design, which is still a coherent description of a batch plus streaming recommendation architecture. The other is running it, which the README describes down to individual hosts and shell commands.

The original framing is a movie recommendation site called "懂你", described as similar in shape to a popular Chinese film database site, where users browse and search films and the site recommends in real time based on browsing records.

## Four deployables, not one: crawler, site, admin, recommender

The system is split into four projects with four different development environments, and the top level directory listing matches that split with `Movie/`, `MovieManager/`, `Spark_Movie/`, `SQL`, `tutorial/` and `zxl_picture/`.

The crawler was built in PyCharm with Python 3.6, on MySQL plus Scrapy. The README states a constraint that dates the project precisely: the crawled content is on the public internet, so a proxy is needed before it will run.

The web site was built with IntelliJ IDEA, Maven, Git, Linux and PowerDesigner, on a MySQL, MyBatis, Spring and SpringMVC stack. It handles browsing, search and rendering recommendations, and the README says it was deployed at a public IP address that readers could visit.

The admin system uses the same toolchain plus EasyUI, and manages user and film information: adding and deleting films and completing user records. Because film images needed somewhere to live, a separate image server was built, with the README pointing to a blog post about setting up FastDFS for that purpose. A test account is published in the README for the admin back end.

That split is the main structural fact. The crawler, the two web applications and the Spark job are separately built artefacts with separate dependencies, and the recommendation system only reaches the web site through a MySQL table.

## The data flow: click events in, a MySQL table of recommendations out

The recommendation system is the interesting part, and the README describes its data flow directly.

The web site is instrumented so that user click events, such as which film a user likes or what they rated a film, are captured and passed to the recommendation system. The recommendation system processes that information and writes the results into the MySQL database, and the web front end queries the database to display recommended films.

The batch side is built on a long stack: Hadoop, ZooKeeper, Flume, Kafka, nginx, Spark, Hive and MySQL. The streaming side reuses nginx and Flume rather than the web application calling the cluster directly.

The role each piece plays is described in the later steps. nginx is installed to receive click information from the movie site and write it to a local file. Flume monitors that local file in real time and sends the data to the Kafka message queue. A Spark Streaming job then consumes from Kafka. For users who are not logged in or are new, a separate class supplies recommendations, defaulting to the five most watched films.

That last detail is a decent piece of engineering honesty. A collaborative filtering model has nothing to say about a user with no history, and the project handles it with a popularity fallback rather than pretending the model generalises.

The final output format is stated as `(userid, movieid, rating)`, stored in MySQL, holding recommended films for all users.

## Four Spark jobs, each submitted by hand

The pipeline is expressed as a sequence of `spark-submit` invocations, and the class names are the clearest documentation of the stages. They are reproduced here as the README prints them.

The first job is the ETL step, in package `com.zxl.datacleaner.ETL`, packaged as `ETL.jar`:

```bash
spark-submit --class com.zxl.datacleaner.ETL --total-executor-cores 2 --executor-memory 2g lib/ETL.jar
```

The README states that this succeeds in creating tables in Hive. The second stage is data preparation and model building, with `com.zxl.datacleaner.RatingData` packaged and uploaded to test the model.

Third, `com.zxl.ml.ModelTraining` is packaged as `Spark_Movie.jar` and builds the model:

```bash
spark-submit --class com.zxl.ml.ModelTraining lib/Spark_Movie.jar
```

Model selection is by RMSE, the root mean square error, and the README notes that a companion class `com.zxl.ml.ModelTraining2` trains a single model with `rank=50`, `iteration = 10`, `lambda = 0.01`. Those three parameters are the only concrete hyperparameters the README gives.

Fourth, `com.zxl.ml.Recommender` produces recommendation results and `com.zxl.ml.RecommendForAllUsers` writes them into MySQL, which needs a JDBC driver passed explicitly:

```bash
spark-submit --class com.zxl.ml.RecommendForAllUsers --jars lib/mysql-connector-java-5.1.35-bin.jar lib/Spark_Movie.jar
```

The streaming job is submitted with the Kafka client jar attached as well:

```bash
spark-submit --class com.zxl.streaming.SparkDrStreamALS --total-executor-cores 2 --executor-memory 1g --jars lib/kafka-clients-0.10.1.0.jar lib/Spark_Movie.jar
```

Note the shape: four separate jar files across the batch path, driver jars inlined with `--jars`, and no scheduler. This is a manual pipeline, run step by step by a person at a terminal.

## Three CentOS 6 hosts with hardcoded IPs, started by hand every time

The deployment section is the part that dates this repository, and it is also the part a reader trying to run it has to reckon with.

Server planning uses a CentOS 6 image with three machines: spark1 at 192.168.13.134 with 8G of memory and 4 cores, spark2 at 192.168.13.135 with 6G and 4 cores, and spark3 at 192.168.13.136 with 6G and 4 cores.

The role assignment is explicit. spark1 runs the namenode, secondary namenode and datanode; spark2 and spark3 run datanodes. YARN puts the resourcemanager and one nodemanager on spark1, with a nodemanager on each of the others. MySQL is installed on spark2, Hive on spark1, and Spark runs in standalone mode with spark1 as master and the other two as workers.

Then every stage has to be started manually before a job runs, and the README prints the exact commands for each:

```bash
start-dfs.sh
start-yarn.sh
```

```bash
service mysqld start
```

```bash
hive --service metastore
```

```bash
./sbin/start-all.sh
```

The later stages add ZooKeeper, Flume and Kafka, each with its own start command, including a Flume agent started with an explicit configuration file and a Kafka server started from `config/server.properties`.

So a single recommendation run is preceded by roughly eight service start commands on two or three machines. That is the honest cost of the architecture as documented, and it is the strongest argument for reading this repository rather than deploying it.

## What the README leaves undocumented

Several things a new user would need are absent, and their absence is informative.

There is no installation command. Every service is installed by following an external blog post: the README links separate CSDN articles for HDFS, YARN, MySQL, Hive, the Spark cluster, HDFS operations, Spark SQL with Hive integration, nginx, Flume, ZooKeeper and Kafka. Eleven separate guides, none of them in this repository.

There is no configuration file content. Host IPs, memory and cores are given, but the HDFS, YARN, Hive and Spark configuration files themselves are not, so those have to be written from the linked guides against the documented topology.

The dataset is named rather than included: the README links MovieLens at grouplens.org and states that the 1m size dataset was downloaded. No schema, no load script, no row counts.

The MySQL side is represented by a `SQL` directory at the repository top level and by the statement that results are stored as `(userid, movieid, rating)`, but the README does not describe the table definitions for the site, the admin system or the crawler store.

There is a `tutorial/` directory in the repository that the README does not describe, and a PowerPoint file at the root that is a defence presentation, the kind of artefact a graduation project produces and a production system does not.

None of this makes the repository wrong. It makes it a project write-up rather than a product.

## Where a Spark recommender is the wrong shape today

The honest case against deploying this is not the algorithm, it is the eleven external guides and three fixed-IP hosts.

The alternative the author themselves chose is the CineMatch rewrite, and the README describes its stack in enough detail to be a real comparison. The new system uses Vue 3 with TypeScript and Vite on the front end, Spring Boot 4 with Java 21 on the back, with Spring Security, MyBatis and Flyway. It supports local H2 runs and MySQL or Docker Compose deployment. It ships with 923 film records and local posters, plus deployment steps, MovieLens data preparation and model training guides, algorithm evaluation and defence materials.

On the modelling side, CineMatch describes a trainable hybrid recommender: ALS-WR matrix factorisation combined with content preferences, neighbourhood collaborative filtering and reputation scores, with diversity re-ranking, and support for training, activating, rolling back versions and exporting. That is a different claim from the RMSE-selected single ALS model here, where the only documented hyperparameters are rank, iteration and lambda.

The structural difference is what matters for adoption. The old design needs Hadoop, YARN, Hive, ZooKeeper, Flume, Kafka, nginx and Spark across three machines. The new one runs on H2 locally or MySQL via Docker Compose.

If you want to study a batch plus streaming Spark recommendation pipeline, this repository is still worth reading, with the caveat that it targets Spark 1.6.1, Kafka 0.10.1.0 and a MySQL 5.1.35 connector, all of which are far behind current releases.

## Conclusion

Adopt this repository as a reference for a full offline-plus-streaming Spark recommendation architecture, or as a study of ALS model selection by RMSE. Do not start new work on it: the README states the project has been rewritten in a separate repository, CineMatch, and that old accounts, historical ratings and the old ALS model do not migrate. If you do run it, the hard part is not the Java but the service bring-up across three CentOS 6 hosts, so verify first that you have the machines with the exact IPs listed.

## FAQ

### What does Movie_Recommend do?

It combines four parts: a Scrapy crawler on MySQL, an SSM movie web site, an EasyUI admin back end, and a Spark recommendation system. Click events from the site are captured and sent to the recommender, whose results are written to MySQL for the front end to display.

### Should I use this repository for a new movie recommendation project?

The README states the project has been modernised in a separate repository, CineMatch, which uses Vue 3, TypeScript, Spring Boot 4 and Java 21 and can run on H2 locally or with Docker Compose. It also notes that old accounts, ratings and the ALS model do not migrate automatically.

### Which algorithm does the recommendation system use?

ALS matrix factorisation from Spark MLLib. Models are selected by RMSE, and the README documents a single-model configuration with `rank=50`, `iteration = 10` and `lambda = 0.01`.

### What servers does the Spark pipeline need?

Three CentOS 6 machines: spark1 at 192.168.13.134 with 8G and 4 cores, spark2 at 192.168.13.135 with 6G and 4 cores, and spark3 at 192.168.13.136 with 6G and 4 cores. HDFS, YARN, Hive, MySQL and standalone Spark are each assigned to specific hosts.

### How are real-time click events collected?

nginx receives the click information from the site and writes it to a local file, Flume monitors that file in real time and sends the data to Kafka, and a Spark Streaming job consumes it. For users who are not logged in or are new, a separate class recommends the five most watched films by default.

### Which dataset does the recommender train on?

MovieLens, linked from grouplens.org, and the README states the 1m size dataset was downloaded. The data is loaded into HDFS, cleaned by an ETL job that creates Hive tables, and processed by the training classes.

## Sources

- [CodeRayZhang/Movie_Recommend on GitHub](https://github.com/CodeRayZhang/Movie_Recommend)
- [Issues](https://github.com/CodeRayZhang/Movie_Recommend/issues)
- [License: MIT](https://github.com/CodeRayZhang/Movie_Recommend/blob/master/LICENSE)
- [README](https://github.com/CodeRayZhang/Movie_Recommend/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/coderayzhang-movie-recommend