Getting Started with OpenSearch

Author

AWS Certified Solutions Architect – Associate

Getting Started with OpenSearch

October 27, 2021

awsopensearchelasticsearch

Getting Started with OpenSearch

Introduction

We are always in search of something. Whether it's the meaning of life or the delicious food near us, we heavily rely on search engines to get the answers. We already use apps such as Swiggy, Uber, Wikipedia, etc., daily. But did you know that these apps use Elasticsearch?

One might ask, why not use the DataBase directly instead of a separate tool? Because when we search for anything, we don't always type the exact words or sentences. For example, we might search for "cinema" and need results for "theater", searching for containers - get results for both box containers and docker containers, etc. In cases like these, the regular database queries will not give the desired result.

OpenSearch(Elasticsearch) is the leading free and open-source search and analytics solution. OpenSearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine and schema-free JSON documents.

There are a lot of tutorials online covering the basics of OpenSearch(Elastic Search). I recommend:

  1. Github: Crash Course to Elastic Stack
  2. Youtube: Beginner’s Crash Course to Elastic Stack
  3. Udemy: Complete Guide to Elasticsearch

Here, we will build a Search Application for Marvel Movies using OpenSearch. We will upload a dataset of marvel movies to the OpenSearch domain and query it using lambda. Our tech stack includes lambda & API-gw for the backend, React for the frontend, and OpenSearch for the database. Check out the Architecture diagram below: architecture image Note: OpenSearch is not a DB alternative, but, since our use case is simple I'm going to use it as DB.

Pre-requisites

  1. Working knowledge of:
    • basic OpenSearch(Elastic-Search) queries
    • ReactJS
    • AWS Lambda
    • AWS API Gateway
  2. Active AWS account

Steps to Create the Application

Step 1: Create the resources for the backend:

1. Open-Search domain

  • Go to OpenSearch console and create a domain.
  • Configuration:
    • Name: marvel-movies // can give any name
    • Deployment type: Development and testing // To avoid large AWS bills we use the smallest instance available
    • Version: 1.0 // Latest version of the tool
    • Data Node: t3.small // this is the cheapest instance, which is enough for our application
    • Network configuration: public // allow access to cluster over internet
    • Enable Fine-grained access control (FGAC): with ‘Create Master User’ // User to login into OpenSearch Dashboard
    • Access policy: Only use fine-grained access control
  • Leave the rest to defaults and Review and create the domain.
    • Note: Domain creation takes around 10-15mins.

2. Lambda function

  • Go to the lambda console and create a new lambda function.
  • Configuration:
    • Function name: get-marvel-movies-from-opensearch
    • Runtime: NodeJS
    • Review and create the function
    • Add an Environment variable DOMAIN_URL, Example: search-marvel-movies-random-string.us-east-2.es.amazonaws.com
    • Note: You can get this url from AWS OpenSearch Console and make sure you remove https:// from Domain_url
    • Go to GitHub repo: repo, and copy-paste the lambda code

3. Now go to the api-gateway console and create an API gateway:

  • Configuration:
    • API type: HTTP API
    • Integration: Add Integration -> select lambda we just created
    • API Name: marvel-movie
    • Configure routes: Method: GET, Resource path: /search, Integration target: lambda function you just created
    • Review and create
  • Go to the API and configure CORS and add ”*” for all the headers, so we dont get any CORS errors. cors
Step 2: Upload the movie data to Open-Search domain
  1. Download the data-set from here and uzip it.
  2. Open terminal and run the following command: curl -XPOST -u '<user-name>:<password>' 'https:// <domain-url> /marvel_movies/_bulk' --data-binary @path/to/file.json -H 'Content-Type: application/json'
  3. Look out for this in the output: data-upload-no-errors
Step 3: Giving Lambda permission to access data on OpenSearch
  • When we have FGAC enabled, we need to map the role in Opensearch domain
  • Go to OpenSearch dashboard by clicking on the link: architecture image
  • Login into domain and open security under OpenSearch Plugins.
  • Select Roles
  • Note: Now either create a new role with appropriate permissions or use a existing role.
  • We will use the existing role all_access, click on it.
  • Now go to Mapped user tab from the top bar and click on Manage mapping.
  • Click on Add another backend role and enter the lambda IAM role ARN.
  • Then click on Map and we have added permissons. opensearch-role-mapping
Step 4: Setting up the front-end react app
  1. Clone the React app on your local system: repo
  2. Run npm i in the app folder
  3. Add the api-gw endpoint url to REACT_APP_API_ENDPOINT in .env file. env-api-url
  4. Finally run npm start
step 5: Once we have everything in place, Let’s test our application
  1. Go to http://localhost:3000 search-result

References

Further Reading

Antstack Blog Post
Antstack Blog Post

Are you planning to go to serverless? AntStack is a cloud computing service and consulting company primarily focusing on Serverless Computing. We help companies get up and running with serverless, and we’ll make sure that there are no limits.

Keep track of our socials and connect with us - LinkedIn