Skip to content

Redis

Definitions

Redis Stack

Redis Stack is a comprehensive package that combines the power of Redis OSS with additional features like probabilistic data structures, queryable JSON documents, and time series data support. It aims to simplify the developer experience and provide a robust platform for real-time application development. RedisInsight is the recommended visualization tool, offering features such as data structure visualization, an advanced command-line interface, real-time traffic analysis, and memory usage analysis. Redis Stack is available in two packages: Redis Stack Server for production deployments and Redis Stack for local development. It supports a wide range of use cases across industries and provides compatibility with popular client libraries like redis-py, node_redis, NRedisStack, Jedis, Redis OM .NET, Redis OM Node, Redis OM Python, and Redis OM Spring.

Redis Stack

Redis OM

Redis OM (Object Mapping) is a set of high-level client libraries for Redis, available in .NET, Node.js, Python, and Java (Spring). It simplifies working with Redis data by providing object mapping and fluent querying capabilities. Developers can define domain objects, persist them in Redis, and retrieve them easily. Redis OM supports serialization as Redis hash or JSON and leverages RedisJSON and RediSearch for efficient indexing and querying. Each library has language-specific features, such as TypeScript support for Node.js and LINQ querying for .NET. Redis OM makes working with Redis and Redis modules more convenient for developers.

Redis OM

Redis OM for Node.js

Redis OM for Node.js is a powerful library designed to simplify the integration of Redis into Node.js applications. It provides intuitive object mapping and a fluent interface, allowing you to seamlessly map Redis data structures to simple JavaScript objects. With Redis OM, you can perform efficient CRUD operations using a repository, making it easy to save, read, and remove entities from Redis.

Key Features

  • Object Mapping: Redis OM enables transparent persistence of domain objects in Redis. You can define schemas that describe the structure of your objects and easily map them to Redis data.
  • Fluent Queries: Redis OM provides a fluent interface for querying Redis data. You can perform searches based on various criteria such as string values, numbers, booleans, dates, and even perform full-text search on text fields. The library takes advantage of RediSearch capabilities to provide efficient and flexible querying.
  • JSON and Hash Support: Redis OM supports both JSON and Hash storage options. By default, entities are stored as JSON documents using RedisJSON. However, you can configure schemas to use Hashes if required, allowing you to choose the most suitable storage method based on your data structure.
  • Sorting and Pagination: Redis OM allows you to sort search results based on specific fields and supports pagination to handle large result sets efficiently. You can sort results in ascending or descending order, providing flexibility in presenting data to your application.
  • Easy Integration: Redis OM seamlessly integrates with Node Redis, a popular Redis client for Node.js. It leverages the power of Node Redis to establish connections, execute Redis commands, and interact with Redis data efficiently.

Redis OM for Node.js

Redis OM for Node.js Guide

Implementation

Redis Stack

Dockerized Application

One of the requirements for this year's Wimma LAB projects is to utilize Docker for further container orchestration in Kubernetes. Thus, the redis/redis-stack image was used.

Docker Image

The container is always built using docker-compose.yml with the following configurations:

  redis-stack:
    image: redis/redis-stack:latest
    container_name: tukko-redis
    restart: always
    ports:
      - 6379:6379
      - 8001:8001
    volumes:
      - ./redis-stack.conf:/redis-stack.conf

Port 6370 is the default port for client connections. Port 8001 allows for access to RedisInsight running inside the container.

RedisInsight

redis-stack.conf is custom configurations that are mounted to the root of the container. The file includes default Redis 6.2 configurations with the following fields changed:

  • protected-mode no
  • save ""
  • requirepass <custom password>
  • maxclients 1
  • maxmemory 250mb

Redis Configurations

Redis OM

Redis OM for Node.js is used for working with Redis Stack. It can be installed with npm install redis-om@beta. Yes, the beta version at this moment, because it includes some new features, such as embedded JSON fields, easier code for clients, and custom IDs for keys in Redis Stack.

Redis OM Beta

Client Setup

src/scripts/redis/client.ts has the code for establishing a connection to Redis Stack via the client. The password defined in the configurations is passed to createClient(). The client is then exported for use by other scripts.

import { createClient } from "redis";

const client = createClient({
  url: (process.env.REDIS_OM_URL || "redis://localhost:6379") as string,
  password: process.env.REDIS_OM_PASSW as string,
});
client.on("error", (err: any) => console.log("Redis Client Error", err));

export { client };

Object Mapping (OM)

Redis uses schemas and repositories to store and manipulate data. Schemas are basically data models, and repositories allow for interaction with a schema stored in it. Thus, schemas are defined in src/models/redis/tmsData.ts, and repositories are created in src/scripts/redis/client.ts and then indexed for efficient searching.

import { Repository } from "redis-om";
import { stationSchema } from "../../models/redis/tmsModels";

const stationRepository = new Repository(stationSchema, client);
await stationRepository.createIndex();

export { stationRepository };

Loading Data

Digitraffic's Endpoints

The main source of data used in Tukko is Digitraffic's API, specifically road traffic endpoints, such as TMS (Traffic measurement system) and Traffic messages.

TMS endpoints:

  • https://tie.digitraffic.fi/api/tms/v1/stations
  • https://tie.digitraffic.fi/api/tms/v1/stations/{id}
  • https://tie.digitraffic.fi/api/tms/v1/stations/data

Traffic messages:

  • https://tie.digitraffic.fi/api/traffic-message/v1/messages

Swagger Documentation

Road Traffic

Note that Digitraffic requires providing headers to identify applications:

const axiosConf = {
  headers: {
    clientName: "WIMMA-lab/IoTitude/Tukko",
  },
};

Fetching and Storing

src/scripts/redis/loadStations.ts, src/scripts/redis/loadSensors.ts, and src/scripts/redis/loadRoadworks.ts are scripts used for fetching data and storing it in Redis Stack. loadSensors.ts will be taken as an example to demonstrate the process of loading data for each schema:

  1. Before fetching, it is checked whether the data is updated in Digitraffic.
  2. Data is fetched from Digitraffic: 
    const response: AxiosResponse = await axios.get(urlData, axiosConf);
  3. New objects are constructed:

function constructSensorObject(sensor: any) {
  return {
    id: sensor.id,
    stationId: sensor.stationId,
    name: sensor.name,
    shortName: sensor.shortName,
    timeWindowStart: sensor.timeWindowStart,
    timeWindowEnd: sensor.timeWindowEnd,
    measuredTime: sensor.measuredTime,
    unit: sensor.unit,
    value: sensor.value,
  };
}
Note that if a property is not additionally defined in a corresponding schema in src/models/redis/tmsData.ts, it will be saved as a nested field, and it will not be possible to query Redis using that property!

  1. Objects are stored in Redis with custom key IDs, and TTL (time to live) is set for them:

await sensorRepository.save(id, sensor);
await sensorRepository.expire(id, calculateTimeToLive(sensor));
5. Some data can be saved as nested fileds or object using json.set(), for example restrictions in road works: 
client.json.set(`roadwork:${id}`, "$.restrictions", restrictions);
More information about this: @redis/json and Redis JSON

Note that the logic might differ for other schemas. For example, before storing stations, old stations are flushed from Redis because TTLs are not set for them.

Scheduling

Data loading scripts are scheduled by the custom scheduler in src/index.ts: 

import { scheduleScript } from "./scripts/schedule";
import { loadStations } from "./scripts/redis/loadStations";
import { loadSensors } from "./scripts/redis/loadSensors";
import { loadRoadworks } from "./scripts/redis/loadRoadworks";

scheduleScript(loadStations, 0, 60000 /* rate=1min */);
scheduleScript(loadSensors, 0, 60000 /* rate=1min */);
scheduleScript(loadRoadworks, 0, 60000 /* rate=1min */);
If Digitraffic data has been updated, the loading process will proceed. Normally, Digitraffic always responds with fresh sensor and roadwork data. However, station data is static and is updated quite rarely.

Retrieving Data

Express Server API

The backend server API is developed using Express. The default port is 3001. The server responds to GET requests via 4 endpoints:

  • https://localhost:3001/stations
  • https://localhost:3001/stations/{id}
  • https://localhost:3001/sensors
  • https://localhost:3001/roadworks

You can find the code for handling requests to endpoints in src/routes/redis/stations.ts, src/routes/redis/sensors.ts, and src/routes/redis/roadworks.ts. There are plenty of options available for building queries. You can see the detailed Swagger documentation via https://localhost:3001/api-docs.

Search Queries

src/scripts/redis/searchStations.ts, src/scripts/redis/searchSensors.ts, and src/scripts/redis/searchRoadworks.ts are scripts used for creating and sending search queries to Redis. src/scripts/redis/searchSensors.ts will be taken as an example to demonstrate this process:

  1. The searchSensors() function is called by the server.
  2. If there are any parameters passed to the function, a dictionary of sensor parameters is generated: buildParamsDict(params, sensorParams);.
  3. A search query is built based on sensor parameters and sent to Redis. Afterwards, filtered data is returned: buildSensorQuery(sensorParamsDict).return.all();.