dev-resources.site
for different kinds of informations.
Commerce Cloud logs in OpenSearch locally
Published at
9/5/2023
Categories
sap
commerce
devops
opensearch
Author
mikolayek
Main Article
Author
9 person written this
mikolayek
open
Issue
Commerce Cloud offers customers to view and search logs in dedicated instance of AWS OpenSearch. Disk quota there is limited, so after some time logs are gone there, but you can find them still available dumped in your dedicated Azure Blob Storage.
You can find that on your Cloud Portal: Environments > [given environment] : Cloud Storage > logs
Unfortunately these files are hard to use as blob entries are impossible to filter by creation date and also by content.
Solution
In order to filter files I have used java code to connect to Azure Blob Storage and then filter roughly by date.
Download Azure Blob Files
In below example I wanted to filter our files which has been created from 11-Aug-2023, 5 AM UTC to 12-Aug-2023 11:59 PM UTC:
package pl.com.like.zip;
import com.azure.identity.*;
import com.azure.storage.blob.*;
import com.azure.storage.blob.models.*;
import java.io.*;
public class BlobLogFetch {
public static void main(String[] args) throws IOException {
var connectionString =
"AccountName=XXXXXXXXXXX;AccountKey=YYYYYYYYYYYYY;EndpointSuffix=core.windows.net;DefaultEndpointsProtocol=https;";
var container = "commerce-logs-separated";
var containerClient = new BlobContainerClientBuilder().connectionString(connectionString)
.containerName(container).buildClient();
System.out.println(containerClient.exists());
var from = OffsetDateTime.of(2023, 8, 11, 5, 0, 0, 0, ZoneOffset.UTC);
var to = OffsetDateTime.of(2023, 8, 12, 23, 59, 0, 0, ZoneOffset.UTC);
containerClient.listBlobs().stream().filter(i -> i.getProperties().getCreationTime().compareTo(from) > 0)
.filter(item -> item.getProperties().getCreationTime().compareTo(to) < 0).forEach(i -> {
var blob = containerClient.getBlobClient(i.getName());
var path = Paths.get(i.getName()).normalize().toAbsolutePath();
// create dirs
path.toFile().getParentFile().mkdirs();
blob.downloadToFile(path.toString(), Boolean.TRUE);
Above code will list all blob files, filter by file properties if meets date range requirements. If creation date is fine, file is downloaded.
Install OpenSearch
Then, we would like to be able to store file content where data are easily searchable.
There are few options on table, but if we have access to SAP Commerce Cloud, we naturally have used Amazon OpenSearch, so we will follow same tooling locally.
It is possible to find OpenSearch on Docker Hub searching for opensearchproject/opensearch.
I have used linked docker-compose.yml as it contains additionally Dashboard.
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-node2:
image: opensearchproject/opensearch:latest
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node2
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:
To run it locally just download file and run:
docker-compose up -d
OpenSearch Configuration
When docker compose is started, you should be able to find OpenSearch Dashboard available on http://localhost:5601/ and logging with credentials admin:admin.
Initially instance will not have any data. We need to create an index first. To do that, click on hamburger menu: OpenSearch Plugins > Index Management, there go to: Index Management > Indices and create index. On my instance I have created index called commerce.
During index creation you can set up data mapping. Below you can find mapping which I have used:
{
"properties": {
"@timestamp": {
"type": "date"
},
"_p": {
"type": "text"
},
"kubernetes": {
"type": "object",
"properties": {
"annotations": {
"type": "object",
"properties": {
"ccv2_cx_sap_com_build-code": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_deployment-id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cni_projectcalico_org_containerID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cni_projectcalico_org_podIP": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cni_projectcalico_org_podIPs": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"data-ingest_dynatrace_com_injected": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"dynakube_dynatrace_com_injected": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"fluentbit_io_parser": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"oneagent_dynatrace_com_injected": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"platform-security-configmap-hash": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"container_hash": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"container_image": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"container_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"docker_id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"host": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"labels": {
"type": "object",
"properties": {
"app_kubernetes_io_component": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"app_kubernetes_io_managed-by": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"app_kubernetes_io_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"app_kubernetes_io_part-of": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_build-code": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_deployment-id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_platform-aspect": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_service-name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"ccv2_cx_sap_com_service-version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"controller-revision-hash": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"controller-uid": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"job-name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"pod-template-hash": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"statefulset_kubernetes_io_pod-name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"update_ccv2_cx_sap_com_mode": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"namespace_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"pod_id": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"pod_name": {
"type": "text"
}
}
},
"log": {
"type": "text"
},
"logs": {
"type": "object",
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"buildDate": {
"type": "date"
},
"container": {
"type": "object",
"properties": {
"image": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"content": {
"type": "object",
"properties": {
"apiToken": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"apiUrl": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"clusterID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"hasHost": {
"type": "boolean"
},
"hostGroup": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"initialConnectRetry": {
"type": "long"
},
"monitoringNodes": {
"type": "object",
"properties": {
"aks-ey4c7ymiac-68098534-vmss000012": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss000013": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss000016": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss000017": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss000019": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"aks-ey4c7ymiac-68098534-vmss00001a": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"networkZone": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"paasToken": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"proxy": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"skipCertCheck": {
"type": "boolean"
},
"tenantUUID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"tlsCert": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"trustedCAs": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"contextMap": {
"type": "object",
"properties": {
"CronJob": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"RemoteAddr": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"Tenant": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"audit": {
"type": "object",
"properties": {
"actionId": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"actionName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"data": {
"type": "object",
"properties": {
"PK": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"changeType": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cronJob": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cronJobResult": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"cronJobStatus": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"gyIsActive": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"gyUID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"idVerified": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"idVerifiedAt": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"job": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"newUser": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"prevUser": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"residenceVerified": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"residenceVerifiedAt": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"typePK": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"user": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"sessionId": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"user": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"bytesSent": {
"type": "long"
},
"commitMillis": {
"type": "long"
},
"connectionStatus": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"myid": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"processMillis": {
"type": "long"
},
"remoteHost": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"requestLine": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"sourceClassName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"sourceMethodName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"statusCode": {
"type": "long"
}
}
},
"endOfBatch": {
"type": "boolean"
},
"env": {
"type": "object",
"properties": {
"containers": {
"type": "object",
"properties": {
"image": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"dataIngestInjected": {
"type": "boolean"
},
"failurePolicy": {
"type": "boolean"
},
"installPath": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"installVersion": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"installerFlavor": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"installerTech": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"installerUrl": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8BasePodName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8BasePodUID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8ClusterID": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8Namespace": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8NodeName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"k8PodName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"mode": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"oneAgentInjected": {
"type": "boolean"
},
"workloadKind": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"workloadName": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"gitCommit": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"goVersion": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"hostTenant": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"instant": {
"type": "object",
"properties": {
"epochSecond": {
"type": "text"
},
"nanoOfSecond": {
"type": "text"
}
}
},
"level": {
"type": "text"
},
"level_value": {
"type": "long"
},
"logger": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"loggerFqcn": {
"type": "text"
},
"loggerName": {
"type": "text"
},
"logger_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"marker": {
"type": "object",
"properties": {
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"message": {
"type": "text"
},
"msg": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"origin": {
"type": "text"
},
"platform": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"thread": {
"type": "text"
},
"threadId": {
"type": "integer"
},
"threadPriority": {
"type": "integer"
},
"thread_name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"thrown": {
"type": "object",
"properties": {
"cause": {
"type": "object",
"properties": {
"cause": {
"type": "object",
"properties": {
"cause": {
"type": "object",
"properties": {
"commonElementCount": {
"type": "long"
},
"extendedStackTrace": {
"type": "object",
"properties": {
"class": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"exact": {
"type": "boolean"
},
"file": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"line": {
"type": "long"
},
"location": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"method": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"localizedMessage": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"message": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"commonElementCount": {
"type": "long"
},
"extendedStackTrace": {
"type": "object",
"properties": {
"class": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"exact": {
"type": "boolean"
},
"file": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"line": {
"type": "long"
},
"location": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"method": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"localizedMessage": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"message": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"commonElementCount": {
"type": "long"
},
"extendedStackTrace": {
"type": "object",
"properties": {
"class": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"exact": {
"type": "boolean"
},
"file": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"line": {
"type": "long"
},
"location": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"method": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"localizedMessage": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"message": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"commonElementCount": {
"type": "long"
},
"extendedStackTrace": {
"type": "object",
"properties": {
"class": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"exact": {
"type": "boolean"
},
"file": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"line": {
"type": "long"
},
"location": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"method": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"localizedMessage": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"message": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
},
"name": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"timeMillis": {
"type": "long"
},
"ts": {
"type": "date"
},
"version": {
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
},
"type": "text"
}
}
},
"record_date": {
"type": "long"
},
"stream": {
"type": "text"
},
"time": {
"type": "date"
}
}
}
Feeding Search Index
Now is the time to upload data from local files to index. Data in blob items is already nicely prepared and stored in JSON format as 1-liner.
Additionally I have added code, which ignores default unsigned SSL certificate, which by default is used locally:
import groovy.json.JsonOutput
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLSession;
import javax.net.ssl.HttpsURLConnection
import javax.net.ssl.SSLContext
import javax.net.ssl.SSLEngine
import javax.net.ssl.SSLSocketFactory
import javax.net.ssl.TrustManager
import javax.net.ssl.X509ExtendedTrustManager
import java.security.cert.CertificateException
import java.security.cert.X509Certificate
import java.util.Base64;
class UnsafeTrustManager extends X509ExtendedTrustManager {
@Override
void checkClientTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}
@Override
void checkServerTrusted(X509Certificate[] x509Certificates, String s, Socket socket) throws CertificateException {}
@Override
void checkClientTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}
@Override
void checkServerTrusted(X509Certificate[] x509Certificates, String s, SSLEngine sslEngine) throws CertificateException {}
@Override
void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}
@Override
void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}
@Override
X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0]
}
}
def handleError(httpResponseException) {
println "Request failed with status code: ${httpResponseException.statusCode}"
println "Error message: ${httpResponseException.message}"
}
def dir = new File('./')
// List all files in the directory
def postmanPost = new URL('https://localhost:9200/commerce/_doc')
def v = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
def sslContext = SSLContext.getInstance("TLS")
sslContext.init(null, new TrustManager[]{new UnsafeTrustManager()}, null)
def sslSocketFactory = sslContext.getSocketFactory()
def userCredentials = "admin:admin"
String basicAuth = Base64.getEncoder().encodeToString(userCredentials.getBytes());
HttpsURLConnection.setDefaultHostnameVerifier(v);
dir.eachFile { file ->
if (file.name.endsWith(".log")) {
println "Sending file ${file.name}"
def inputFile = new File(dir, file.name);
// Read file line by line
inputFile.eachLine { line ->
try {
HttpsURLConnection conn = (HttpsURLConnection) postmanPost.openConnection();
HttpsURLConnection.setDefaultHostnameVerifier(new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
});
conn.setSSLSocketFactory(sslSocketFactory);
conn.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
conn.setRequestProperty ("Authorization", "Basic ${basicAuth}");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("POST");
conn.setHostnameVerifier(v);
def os = conn.getOutputStream();
os.write(line.trim().getBytes("UTF-8"));
os.close();
if (conn.getResponseCode() >= 300) {
println("Failed : HTTP error code : ${conn.getResponseCode()} for ${line}");
}
} catch(Exception e) {
println("Error: ${e}, for line ${line}");
}
}
}
}
Run script by calling:
groovy [script-name].groovy
After that, you will be able to see data in you Dashboard.
commerce Article's
30 articles in total
Top 10 M.Com Project Topics & Ideas For Students
read article
Key Benefits of Choosing Headless Commerce
read article
Enhance Retail with Composable Commerce for B2C
read article
Reducing Delivery Times and Costs: How Machine Learning Optimizes Delivery Routes Efficiently
read article
Advance Features and Deploying the Project (Nerd Streetwear Online Store) Part III
read article
Combine proxies to boost e-commerce results
read article
Next-Generation No-Code E-commerce Website Builder
read article
Coinbase Commerce: A Complete Guide to Seamless Cryptocurrency Payments
read article
The Dynamics of Composable Commerce: The Path to Agile and Scalable Business Solutions
read article
Marketplace vs. Own E-commerce Website: Where Should You Sell?
read article
Commerce Cloud logs in OpenSearch locally
currently reading
Testing and Quality Assurance for E-Commerce Websites
read article
SAP Commerce Cloud and Broken Smart Edit
read article
SAP Commerce Cloud and Read-Only Replica
read article
The 5 Best Importance of B2B Ecommerce for Your Business
read article
Commerce Cloud Exporting Integration Object using Delta Detection
read article
Medusa Vs Woocommerce: Comparing Two Open source Online Commerce Platforms
read article
Getting started with Shoket
read article
SAP Commerce Cloud: 10 Things You Should Know
read article
What Are The World-Class Features of Adobe Commerce Development?
read article
Principles and Practice of Accounting By CA/CMA Santosh Kumar
read article
Deploy Sylius to Heroku
read article
What Makes a Great Checkout Experience?
read article
Build your next commerce store with SvelteKit
read article
How to add new users to the Magento Cloud panel?
read article
Working with the Commerce Engine & Docker in Sitecore Experience Commerce 10.1
read article
Why go Headless?
read article
can i build ecommerce by react and node ? U have suggest for me ? Thank for share
read article
Headless eCommerce vs Traditional eCommerce
read article
Update all items that share the same catalog entity when the item is saved
read article
Featured ones: