Heygidi: Visualizing Two Decades of Turkish News Headlines with Elasticsearch and d3.js

What Heygidi Solves and Who It Serves

Heygidi addresses a narrow but real problem: how to turn a large set of Turkish news headlines into a visual summary of recurring words over a 20-year span. The README states it uses the Hurriyet API to fetch headlines from the Gündem category, from 1997 to 2017. It is aimed at people who want to see which terms dominated Turkish news agenda across those years. The intended audience is likely a developer, data journalist, or researcher who already has some infrastructure knowledge, because the project is not a hosted service. It is a self-contained analysis pipeline that you run locally. The output is a React application that displays the word frequencies, but the core value is the backend scripts that handle data collection, indexing, and aggregation.

The Pipeline: From API to Visualization

The project is split into a backend and a frontend. The backend uses four Node.js scripts in the `backend/bin` directory. First, `fetchData.js` pulls raw headline data from the Hurriyet API. Second, `createMapping.js` sets up the Elasticsearch index mappings. Third, `indexData.js` loads the fetched data into Elasticsearch. Fourth, `exportData.js` counts the words and exports the results. The README notes that zemberek-nlp is used to find the roots of Turkish words, which is necessary because Turkish is an agglutinative language with many suffixes. Elasticsearch performs the counting, and d3.js handles the visualization. The data flow is linear: raw JSON from the API, stored and indexed in Elasticsearch, then aggregated and sent to the React frontend. This is a classic ETL pattern, and the separation of steps makes it possible to rerun only the failing stage.

Getting It Running: Configuration and Scripts

To run Heygidi, you must first copy `backend/config.json.example` to `backend/config.json` and fill in your own credentials. The README is explicit: you need to update the config file with your information before running any commands. The required steps are: run `backend/bin/fetchData.js` to fetch data from the Hurriyet API, run `backend/bin/createMapping.js` to create Elasticsearch mappings, run `backend/bin/indexData.js` to transfer the fetched data into Elasticsearch, and run `backend/bin/exportData.js` to count the words. The React application lives in the `app` directory. There is no mention of a package manager or a single command to start everything, so you will likely run `npm install` in both the backend and app directories, but the README does not confirm that. The scripts are separate, which gives you control but also means you must orchestrate them manually.

A Genuine Limitation: The Missing Data and Stale API

The most obvious limitation is that the project depends on the Hurriyet API, which may no longer provide free or open access as it did when the project was created. The README does not mention any API key expiry or rate limits, but any change to the API's terms or endpoint structure would break `fetchData.js`. Also, the data range ends in 2017, so the visualization is historical, not current. If you want to analyze recent headlines, you would need to modify the date range in the fetch script, but the README does not document where that range is configured. Another limitation is that the project assumes you have Elasticsearch installed and running. It does not provide a Docker setup or a schema migration tool, so you must handle version compatibility yourself. The README does not list the required Elasticsearch version, which could cause unexpected errors.

The Alternative: Using a General-Purpose Text Analysis Stack

A direct alternative to Heygidi is to build a similar pipeline using Python with libraries like pandas, scikit-learn, and a Turkish stemmer such as SnowballTurkish or Zemberek-Python. The difference in approach is that Heygidi uses Elasticsearch as the counting engine, which is a distributed search engine, while a Python script could load the headlines into memory and use collections.Counter for frequency counting. For a dataset of a few thousand headlines, Elasticsearch is overkill. The alternative would be simpler to debug and would not require a running Elasticsearch cluster. However, Heygidi's approach scales better if you plan to analyze millions of documents, because Elasticsearch can handle large aggregations efficiently. The trade-off is operational complexity versus raw counting simplicity. If you already run Elasticsearch for other reasons, Heygidi fits naturally; if not, a Python script is lighter.

Maintenance and Upgrade Cost

The project is archived? No, the repository is not archived, but there are no recent releases and the last push is unknown. The README is in Turkish and appears to be written once, with no update notes. The dependency on zemberek-nlp, which is a Java library, means you need a Java runtime or a bridge, though the README does not explain how it is integrated. The code is from around 2017, so the JavaScript may use older syntax or libraries that have since been deprecated. Upgrading to a modern Elasticsearch version (8.x) likely requires changes to the mapping and query syntax. The MIT license gives you freedom to modify and redistribute, but you must keep the copyright notice. There is no contribution guide or issue tracker visible in the material, so community support is minimal. In short, expect to spend time on maintenance if you adopt this project.

What the Visualization Actually Shows

The README describes the output as a visualization of the most used words in Gündem headlines. It does not specify the exact chart type, but d3.js is known for flexible SVG-based charts, so it could be a bar chart, a word cloud, or a time series. The project's homepage (https://alpcanaydin.github.io/heygidi) likely hosts a live demo, but the README does not describe the interactive features. The core output is a React app, which suggests a single-page interface where you can explore word frequencies by year or across the whole period. The use of zemberek-nlp for stemming means the words are normalized to their roots, so you will see 'deprem' instead of 'depremde' or 'depremin'. This is essential for meaningful frequency counts in Turkish. Without stemming, the counts would be fragmented by suffixes, making the visualization noisy. This is a thoughtful design choice.

Editorial conclusion

Adopt Heygidi if you are a developer or researcher comfortable with Node.js, Elasticsearch, and Turkish text processing, and if you want a reproducible pipeline for analyzing a specific news archive. Do not use it if you expect a ready-made dashboard or if you lack a Hurriyet API key, because the data fetch step will fail. Before running, verify that your Elasticsearch version is compatible with the client library used in the scripts, and check that your API key has access to the historical Gündem articles. The project is a solid starting point for similar corpus analysis, but it is a single-purpose tool, not a general text analytics framework.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository