BiSheServer: a Django and Spark movie recommendation system with user profiles

What the project is, and what it is not

BiSheServer is described plainly as an undergraduate graduation project, with the title of the thesis being the design and implementation of a user-profile based movie recommendation system. The author invites people to continue improving it and says issues can be filed or sent by email or QQ.

So it should be read as a reference implementation rather than a product. That is not a criticism, it is the useful framing. A graduation project that documents the whole chain in one repository, with an architecture diagram, a module diagram, a directory tree and a configuration file annotated line by line, is a better teaching artefact than most architecture write-ups.

The stack is described in the same paragraph. Django is the base framework, in MTV style, with MongoDB, MySQL and Redis as the databases. The base data source is movie data crawled from the Douban platform. User tags are developed mainly from the user's basic information and from behaviour information such as operation records. Hadoop and Spark do the analysis and processing. The management system uses Django's built-in admin, restyled with simpleui.

The repository is Apache-2.0 licensed, and the author asks that attribution be given for quotation and reprinting. The last push was on 2026-06-02, and there are no tagged releases.

The directory tree tells you where the three databases are used

The repository layout is the clearest description of the architecture, and the comments in the tree are unusually informative.

The Django side is split into a BiShe app for the page views, the admin configuration and the URLs, a BiSheServer package holding settings.py and the root URL configuration, and separate movie and user apps each with models.py, views.py and urls.py, where the models map to database tables and the views are the API interfaces.

The api/ directory is the interesting one, because it holds the pieces that are not really CRUD. There is a captcha module, an email module with a separate email_vail module for pre-send verification, a districts module that queries regional addresses, a redis_pool for the Redis connection pool, a model_json module that converts query results, a response module that unifies the API response format, a middleware_log custom logging middleware, a MidnightRotatingFileHandler for thread-safe log rotation, a delay_work module that uses threads to modify tags, a crontab module for scheduling, an upload_log module for uploading log files, and views.py for the 404 and 500 error pages.

A config/ directory holds conf.ini, the main configuration file, and a spark/ directory holds the MySQL connection jar and a spark.py that processes user profiles and can run independently.

Reading conf.ini, where every service endpoint is declared

The configuration is kept out of settings.py and in an ini file, which is a sensible choice for a project that will be installed on several machines, and the file is annotated in Chinese with a comment per line.

The DEFAULT section holds the general switches and paths:

    DEBUG = True
    USE_LOG = False
    ALLOWED_HOSTS = [*]
    STATIC_HOME = static
    LOG_HOME = log/
    TEMPLATES_HOME = templates

The remaining sections are the service connections, each marked as required or optional. The MySQL section sets the engine to django.db.backends.mysql and holds the database name, host, port, user and password. The Redis section sets the host, port, password and the database number. The EMAIL section is optional and includes a switch, EMAIL_USE, which when False means no real verification mail is sent and the code simply prompts for the captcha. The HADOOP_LOG section holds the Hadoop host, the client directory, the remote path on HDFS, the local path and a log suffix format that doubles as the slicing rule.

Note that the section name for the database is spelled DATEBASE rather than DATABASES, and that the file ships with a populated password. Both are things to fix on installation rather than features.

The install path, from migrations to a running server

The installation notes are in a separate document, Install_And_Start.md, and the sequence is documented in the readme.

First the tables and the admin account are created with the standard Django commands, run in the project directory:

python manage.py makemigrations user movie api
python manage.py migrate
python manage.py createsuperuser

Then the base data is downloaded from a Baidu Pan link given in the documentation, and imported by logging into the MySQL console and using the source command against the SQL files:

mysql> source sys.sql

With that in place the service starts with:

python manage.py runserver 0.0.0.0:8001

The documentation explains the address argument, that 8001 is the port and can be changed, that 0.0.0.0 means all hosts can reach it while 127.0.0.1 means local access only.

The import order is not arbitrary and is called out in the notes. All movie data is based on the movie_collectmoviedb table and user data on user_usersbase, and these two must be imported first, with the rest following, and on deletion the order is reversed, deleting dependent rows first. The notes also record that the database needs MySQL 5.6 or newer, that Python should be 3.7, that the MySQL character set must be UTF-8, and that the account with id 2 in user_usersbase is the shared guest account and must not be removed.

The offline path: a PySpark job and a nightly cron

The recommendation computation is deliberately not in the request path. The documentation states that Hadoop and Spark are for offline recommendation analysis, and that if you have no such requirement you can turn them off, and that the whole system runs without installing them.

Setting it up takes two steps. The MySQL connector jar in the spark/jars directory is copied into the Spark jars directory, and the spark.py file is edited for its Spark and MySQL configuration, then uploaded to a Linux machine and added to the scheduled task list so it runs daily:

python3 ./spark.py

The description of what that job does is worth reading, because it explains the recommendation model. The daily offline processing produces recommendations that combine two methods, a content-based approach based on the movies themselves, and a collaborative filtering approach based on user similarity. The documentation also notes that later work could extend this to real-time recommendation.

The separation is the design point. The Django application serves users and records behaviour, and the batch job reads the accumulated data and recomputes the recommendations offline. The cost of that choice is freshness, since a recommendation is only as current as the last nightly run, and the benefit is that the web tier never waits on a Spark job.

The dependency list pins a 2019 stack

The requirements file is the clearest single indicator of the project's vintage, and every version is pinned exactly.

Django is at 3.0.3, with djangorestframework at 3.11.0, django-simpleui at 2021.4.3, django-apscheduler at 0.5.2, django_debug_toolbar at 3.2 and django-extensions at 3.1.3. PySpark is at 3.1.1 with findspark at 2.0.1 and PyHDFS at 0.3.1. The storage layer brings PyMySQL at 1.0.3 and redis at 3.4.1. Around them sit APScheduler, yagmail, requests, pandas, numpy, beautifulsoup4, Pillow, snownlp, phone, botocore and a duplicated pandas entry.

Django 3.0.3 with Python 3.7 is a combination from 2019 and 2020, and the notes confirm Python 3.7 is the target. That is fine for reading the code and running the documented steps, and it is not fine for anything you intend to deploy, since those versions no longer receive fixes.

One entry in the file is a genuine trap rather than an old version. The common errors section shows an ImproperlyConfigured exception saying mysqlclient 1.3.13 or newer is required, and the documented workaround is to find the two lines of code that check the version, comment them out with the # character, and ignore the error. Commenting out a version check is a way to get past the symptom, and the actual fix is to install a mysqlclient that meets the requirement.

Where this project is useful and where it is not

There is a clear line here, and it is worth stating before anyone spends a weekend on it.

The project is useful as a complete worked example. It shows how a user profile is derived from behaviour data, how tags are updated in a background thread, how a Spark job reads the accumulated records from MySQL and writes back recommendations, and how a Django front end consumes them. It also includes the parts that are easy to forget, the captcha, the email verification switch, the Redis pool, the log rotation handler, the unified response format and the error pages, and it includes the base data so you do not have to crawl Douban yourself.

It is not useful as a foundation for a production recommendation service, and the reasons are visible rather than hidden. The configuration ships with a populated MySQL password and a sample account with known credentials, which the notes tell you to change but which is a bad default to leave in a repository. The dependencies are years old. The base data is a crawl of a third party platform, which is both a licensing question and a freshness question. The batch pipeline is a cron job on a single Linux host, so there is no scheduling resilience, no retry policy and no monitoring.

The author also links a related project, recommenderSystemBasedOnSpark, as a reference. The framing of the project as a graduation design, with a thesis title and an invitation to improve it, is the honest summary: this is a document of how one student built the thing, and it is a good document.

Editorial conclusion

Adopt BiSheServer if you are studying how a recommendation service is put together end to end, because it shows the full chain from crawled movie data through user profiling to a nightly Spark batch and back to the web tier, with every layer readable in one repository. Do not adopt it as production software, since it pins Python 3.7, Django 3.0.3, a MySQL account with a plaintext password in the config file, and a sample account with known credentials. Before you run it, change every default credential in config/conf.ini, create the MySQL schema with utf-8 as the documentation insists, and read the note about the user_usersbase guest account with id 2, which the project states must not be removed.

Frequently asked questions

Official sources

1. HuQi2018/BiSheServer on GitHub
2. Issues
3. License: Apache-2.0
4. README