October 26, 2024
AWS OpenSearch is a highly scalable open-source platform for search and analytics, It has a broad range of use cases. Data can be ingested from various sources and quickly searched, analyzed, and visualized. Prominent use cases include log analytics, application monitoring, anomaly detection, and website search.
OpenSearch and OpenSearch Dashboards were originally derived from Elasticsearch 7.10.2 and Kibana 7.10.2. The software started in 2021 as a fork of Elasticsearch and Kibana, with development led by Amazon Web Services. The goal is to make OpenSearch a genuine community-driven application with several different maintainers. It has a set of plugins that further extend the capabilities of OpenSearch.
Log Analytics and Observability Analyze logs generated by websites, applications, and systems. Correlate logs, metrics, and traces to gain insights. Machine learning driven anomaly detection. Use cases include performance and security monitoring, auditing, alerting, and threat detection.
Full-text Search Lightning fast and real-time search at scale, relevant results from semi-structured and unstructured data.
Use cases include:
We have the following states in OpenSearch, but it is not mandatory to move data through all of these states:
Git Repository: https://github.com/Huzaifa-Asif/Open-Search-Tutorial
Git Repository contains the code that is explained below in this article.
To Run the OpenSearch Service Locally, the best option is to use Docker for it, so in case you do not have Docker installed in your system then go to the provided link to download it first.
Link: https://www.docker.com/products/docker-desktop/
In the configuration file, we have two nodes of OpenSearch that will run on port 9200. Then we have OpenSearch Dashboard to visualize and query the data that will run on port 5601.
docker-compose.yaml
version: '3'
services:
opensearch-node1: # This is the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
image: opensearchproject/opensearch:latest # Specifying the latest available image - modify if you want a specific version
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster # Name the cluster
- node.name=opensearch-node1 # Name the node that will run in this container
- discovery.seed_hosts=opensearch-node1,opensearch-node2 # Nodes to look for when discovering the cluster
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2 # Nodes eligible to serve as cluster manager
- bootstrap.memory_lock=true # Disable JVM heap memory swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
ulimits:
memlock:
soft: -1 # Set memlock to unlimited (no soft or hard limit)
hard: -1
nofile:
soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
ports:
- 9200:9200 # REST API
- 9600:9600 # Performance Analyzer
networks:
- opensearch-net # All of the containers will join the same Docker bridge network
opensearch-node2:
image: opensearchproject/opensearch:latest # This should be the same image used for opensearch-node1 to avoid issues
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node2
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
networks:
- opensearch-net
opensearch-dashboard:
image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
container_name: opensearch-dashboard
ports:
- 5601:5601 # Map host port 5601 to container port 5601
expose:
- "5601" # Expose port 5601 for web access to OpenSearch Dashboards
environment:
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:
docker compose up
URL: http://localhost:5601/
Username: admin | Password: admin
var express = require('express');
var app = express();
const { Client } = require('@opensearch-project/opensearch');
const dotenv = require('dotenv');
dotenv.config({ path: '.env' });
const client = new Client({
node: "https://admin:admin@localhost:9200"
});
const index_name = 'books';
// route to create index
app.get('/create-index', async (req, res) => {
try {
const settings = {
settings: {
index: {
number_of_shards: 1,
number_of_replicas: 1,
},
},
};
const response = await client.indices.create({
index: index_name,
body: settings,
});
res.send(response);
} catch (e) {
res.status(500).json(e);
}
});
// route to add document
app.get('/add-document', async (req, res) => {
try {
const document = {
title: 'The Outsider',
author: 'Stephen King',
year: '2018',
genre: 'Crime fiction',
};
const response = await client.index({
index: index_name,
body: document,
refresh: true,
});
res.send(response.body);
} catch (e) {
res.status(500).json(e);
}
});
// route to search document
app.get('/search-document', async (req, res) => {
try {
var query = {
query: {
match: {
title: {
query: 'The Outsider',
},
},
},
};
var response = await client.search({
index: index_name,
body: query,
});
res.send(response.body.hits);
} catch (e) {
res.status(500).json(e);
}
});
// route to search document
app.get('/delete-document', async (req, res) => {
try {
var response = await client.delete({
index: index_name,
id: '1',
});
res.send(response.body);
} catch (e) {
res.status(500).json(e);
}
});
// route to delete index
app.get('/delete-index', async (req, res) => {
try {
var response = await client.indices.delete({
index: index_name,
});
res.send(response.body);
} catch (e) {
res.status(500).json(e);
}
});
app.listen(3000, () => {
console.log('Express app listening on port 3000!');
});
{
"name": "opensearch-nodejs",
"version": "1.0.0",
"description": "test project to demonstrate the functionality of opensearch ",
"main": "app.js",
"scripts": {
"start": "node app.js"
},
"repository": {
"type": "git",
"url": "https://github.com/Huzaifa-Asif/Open-Search-Tutorial.git"
},
"keywords": [
"opensearch",
"opensearch dashboard",
"nodejs"
],
"author": "Huzaifa Bin Asif <huzaifa8580@gmail.com>",
"license": "ISC",
"bugs": {
"url": "https://github.com/Huzaifa-Asif/Open-Search-Tutorial/issues"
},
"homepage": "https://github.com/Huzaifa-Asif/Open-Search-Tutorial#readme",
"dependencies": {
"@opensearch-project/opensearch": "^2.2.0",
"dotenv": "^16.0.3",
"express": "^4.18.2"
}
}
npm install
npm run start
https://github.com/Huzaifa-Asif/Open-Search-Tutorial
