Piemērs sagatavots, izmantojot šādu konfigurāciju:
uname -a
Linux janqis 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
kubectl version
Client Version: v1.31.5
Kustomize Version: v5.4.2
Server Version: v1.32.1
helm version
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}
https://minikube.sigs.k8s.io/docs/start/?arch=%2Flinux%2Fx86-64%2Fstable%2Fbinary+download
curl -LO https://github.com/kubernetes/minikube/releases/latest/download/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64
minikube start
Izveido “dagr” mapi.
“dagr” mapē izveido “simple” mapi (2.1. nodaļas ietvarā turpmāk kā darba mape).
Darba mapē ievieto datni dagr-connector.jar.
Datni var lejupielādēt no DAGR serveriem, piemēram: https://test.dagr.gov.lv/download/dagr-connector
Darba mapē ievieto “application.conf”. Fails definē darbības mainīgos un parametrus dagr-connector saslēgšanai ar DAGR serveri
dataset_names = random # datu kopu (servisu) nosaukumi DAGR sistēmā
datasource_type = random
target_server_uri = "https://dev.dagr.gov.lv" # DAGR serveris, kas saņems datus
provider.key = "MIICeAIBAD..."
provider.client_id = "TLS_TEST_1"
jwt.service.uri = "https://dev.dagr.gov.lv/jwt/token"
jwt.payload-audience = "urn:DEV-LNB:DAGR.STS/connect/token"
metadata_file = "/app/connector.yaml"
# If "pekko.quartz.schedules.connector" is set to null, connector will run once and exit
pekko.quartz.schedules.connector = null
# Šī sadaļa attiecas tikai uz testēšanu ----------------------------------------
random.max_key = 9999
random.batch_size = 3
random.batch_interval = 67s
Darba mapē ievieto “connector.yaml”. Fails definē datu struktūru.
Piemērā tukšs fails, bet svarīgi failu definēt, jo application.conf konfigurācijā ir atsauce uz to metadata_file = "/app/connector.yaml"
Darba mapē ievieto “logback.xml”. Fails definē žurnālfaila veidošanas īpašības.
<configuration scan="true" scanPeriod="15 seconds" >
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d %highlight(%-5level) %-15logger{15} -%kvp- %msg%n%ex{0}</pattern>
</encoder>
</appender>
<logger name="com.zaxxer.hikari" level="warn"/>
<logger name="dagr.data" level="info"/>
<logger name="dagr.http" level="warn"/>
<logger name="dagr.jdbc" level="warn"/>
<logger name="dagr.jdbc.sql" level="warn"/>
<logger name="org.quartz" level="warn"/>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
Darba mapē ievieto “start-dagr-connector.sh”. Fails definē dagr-connector aplikācijas palaišanas secību doker image.
#!/usr/bin/env bash
java -server -Dfile.encoding=UTF-8 -Dconfig.file=/app/application.conf -Dlogback.configurationFile=/app/logback.xml -cp ‘.:/app/:/app/*‘ dagr.Connector
Darba mapē izveido “dagr-connector.dockerfile”. Fails definē darbību kopu, kas nepieciešams, lai izveidotu docker image.
FROM eclipse-temurin:17.0.14_7-jre-focal
COPY dagr-connector.jar /app/
COPY connector.yaml /app/
COPY application.conf /app/
COPY logback.xml /app/
COPY start-dagr-connector.sh /app/start-dagr-connector.sh
RUN chmod +x /app/start-dagr-connector.sh
WORKDIR /app
CMD [ "bash", "/app/start-dagr-connector.sh" ]
Dotās darbības rezultātā, tiks izveidots lokāls docker image, kas publicēts un pieejams lokālā datora ietvarā, pēc tam docker image tiek tieši palaists Kubernetes vidē (bez deployment).
cd ~/dagr/simple
eval $(minikube docker-env)
docker build -t dagr-connector-simple:1.0.1 -f dagr-connector.dockerfile .
kubectl run dagr-connector --image=dagr-connector-simple:1.0.1 --image-pull-policy=Never
kubectl get pods
kubectl logs dagr-connector --tail 1000
kubectl delete pod dagr-connector
Nodaļā 2.1. dagr-connector tika palaists kā Kubernetes pod, šajā nodaļā tiks demonstrēts ilglaicīgāks risinājums, kur dagr-connector tiks palaists kā Kubernetes deployment, joprojām ar iebūvētu (nemaināmu) konfigurāciju
Kā iepriekšējā 2.1. nodaļā sākot no 2.1.1. līdz 2.1.7.
“dagr” mapē sagatavo deployment manifestu dagr-connector.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: dagr-connector
name: dagr-connector
# namespace:
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dagr-connector
template:
metadata:
labels:
app.kubernetes.io/name: dagr-connector
spec:
containers:
- image: "dagr-connector-simple:1.0.1"
imagePullPolicy: Never
name: dagr-connector
cd ~/dagr
kubectl apply -f dagr-connector.yaml
kubectl get pods
kubectl logs deployment/dagr-connector-random --tail 1000
kubectl delete -f dagr-connector.yaml
Salīdzinājumā ar 2.2. piemēru
Sagatavo docker image:
FROM eclipse-temurin:17.0.14_7-jre-focal
COPY dagr-connector.jar /app/
COPY start-dagr-connector.sh /app/start-dagr-connector.sh
RUN chmod +x /app/start-dagr-connector.sh
WORKDIR /app
CMD [ "bash", "/app/start-dagr-connector.sh" ]
cd ~/dagr/docker-image
eval $(minikube docker-env)
docker build -t dagr-connector:1.1.1 -f dagr-connector.dockerfile.
Iepriekš docker image iebūvēto konfigurācijas failu pārnešana uz manifesta failu
apiVersion: v1
kind: Namespace
metadata:
name: dagr
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: dagr-connector
name: dagr-connector
namespace: dagr
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dagr-connector
template:
metadata:
labels:
app.kubernetes.io/name: dagr-connector
spec:
containers:
- image: "dagr-connector:1.1.1"
imagePullPolicy: Never
name: dagr-connector
volumeMounts:
- name: config-volume
mountPath: /app/application.conf
subPath: applicationConf
- name: config-volume
mountPath: /app/connector.yaml
subPath: connectorYaml
- name: config-volume
mountPath: /app/logback.xml
subPath: logbackXml
volumes:
- name: config-volume
configMap:
name: dagr-connector-config-files
---
apiVersion: v1
kind: ConfigMap
metadata:
name: dagr-connector-config-files
namespace: dagr
data:
applicationConf: |
dataset_names = random # datu kopu (servisu) nosaukumi DAGR sistēmā
datasource_type = random
target_server_uri = "https://dev.dagr.gov.lv" # DAGR serveris, kas saņems datus
provider.key = null # "MIICeAIBAD..."
provider.client_id = "TLS_TEST_1"
jwt.service.uri = "https://dev.dagr.gov.lv/jwt/token"
jwt.payload-audience = "urn:DEV-LNB:DAGR.STS/connect/token"
metadata_file = "/app/connector.yaml"
# If "pekko.quartz.schedules.connector" is set to null, connector will run once and exit
pekko.quartz.schedules.connector = null
# Šī sadaļa attiecas tikai uz testēšanu ----------------------------------------
random.max_key = 9999
random.batch_size = 3
random.batch_interval = 67s
connectorYaml: |
#empty
logbackXml: |
<configuration scan="true" scanPeriod="15 seconds" >
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d %highlight(%-5level) %-15logger{15} -%kvp- %msg%n%ex{0}</pattern>
</encoder>
</appender>
<logger name="com.zaxxer.hikari" level="warn"/>
<logger name="dagr.data" level="info"/>
<logger name="dagr.http" level="warn"/>
<logger name="dagr.jdbc" level="warn"/>
<logger name="dagr.jdbc.sql" level="warn"/>
<logger name="org.quartz" level="warn"/>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
cd ~/dagr
kubectl apply -f dagr-connector-with-config-map.yaml
namespace/dagr created
deployment.apps/dagr-connector created
configmap/dagr-connector-config-files created
kubectl get pods -n dagr
kubectl get cm -n dagr
kubectl logs -n dagr deployment/dagr-connector
kubectl delete -f dagr-connector-with-config-map.yaml
Darba mapē ievieto dagr-connector.jar
datni var izgūt no DAGR servera, piem., https://dev.dagr.gov.lv/download/dagr-connector
Darba mapē ievieto “dagr-start-connector.sh”. Fails definē dagr-connector aplikācijas palaišanas secību doker image
#!/usr/bin/env bash
cp /app/conf/application.conf /app/application.conf
cp /app/conf/connector.yaml /app/connector.yaml
cp /app/conf/logback.xml /app/logback.xml
ls -l >> /proc/1/fd/1
java -server -Dfile.encoding=UTF-8 -Dconfig.file=/app/application.conf -Dlogback.configurationFile=/app/logback.xml -cp /opt/jdbc-drivers/postgresql-42.7.4.jar:/app/dagr-connector.jar dagr.Connector
Pievērst uzmanību:
ls -l >> /proc/1/fd/1 izdrukā mapes “app” saturu, lai pārliecinātos, ka visi faili eksistē
Darba mapē izveido “dagr-connector.dockerfile”. Fails definē darbību kopu, kas nepieciešams, lai izveidotu docker image.
FROM eclipse-temurin:17.0.14_7-jre-focal
RUN apt-get update && apt-get install nano -y
RUN rm -rf /var/lib/apt/lists/* && apt-get clean
RUN cd /opt && mkdir jdbc-drivers
RUN wget -O /opt/jdbc-drivers/postgresql-42.7.4.jar https://jdbc.postgresql.org/download/postgresql-42.7.4.jar
COPY dagr-connector.jar /app/
COPY start-dagr-connector.sh /app/start-dagr-connector.sh
RUN chmod +x /app/start-dagr-connector.sh
WORKDIR /app
CMD [ "bash", "/app/start-dagr-connector.sh" ]
Pievērst uzmanību:
cd ~/dagr/docker-image
eval $(minikube docker-env)
docker build -t dagr-connector-example:3.3.3 -f dagr-connector.dockerfile --no-cache .
provider.key = ${PROVIDER_PRIVATE_KEY}
provider.client_id = ${PROVIDER_CLIENT_ID}
username = ${DB_USER}
password = ${DB_PASSWORD}
apiVersion: v1
kind: Namespace
metadata:
name: dagr
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: dagr-connector
name: dagr-connector
namespace: dagr
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dagr-connector
template:
metadata:
labels:
app.kubernetes.io/name: dagr-connector
spec:
containers:
- image: "dagr-connector-example:1.1.1"
imagePullPolicy: Never
name: dagr-connector
envFrom:
- secretRef:
name: dagr-connector-secrets
volumeMounts:
- name: config-volume
mountPath: /app/conf/application.conf
subPath: applicationConf
- name: config-volume
mountPath: /app/conf/connector.yaml
subPath: connectorYaml
- name: config-volume
mountPath: /app/conf/logback.xml
subPath: logbackXml
volumes:
- name: config-volume
configMap:
name: dagr-connector-config-files
---
apiVersion: v1
kind: ConfigMap
metadata:
name: dagr-connector-config-files
namespace: dagr
data:
applicationConf: |
dataset_names = [ # datu kopu (servisu) nosaukumi DAGR sistēmā
sql_sample,
]
datasource_type = jdbc
target_server_uri = "https://dev.dagr.gov.lv" # DAGR serveris, kas saņems datus
provider.key = ${PROVIDER_PRIVATE_KEY}
provider.client_id = ${PROVIDER_CLIENT_ID}
jwt.service.uri = "https://dev.dagr.gov.lv/jwt/token"
jwt.payload-audience = "urn:DEV-LNB:DAGR.STS/connect/token"
metadata_file = "/app/connector.yaml"
pekko.quartz.schedules.connector.expression = "*/10 * * ? * *"
jdbc {
data {
jdbcUrl = "jdbc:postgresql://xxx.yyy:5432/any"
username = ${DB_USER}
password = ${DB_PASSWORD}
autoCommit = false
}
}
connectorYaml: |
name: sql_sample
key: dname
table:
query: >-
with
depts(dname, created) as (values
('sales', '2024-11-14'),
('management', '2024-11-11')
),
emps(dname, ename) as (values
('sales', 'Blake'),
('sales', 'Scott'),
('management', 'King')
),
emps_grouped_by_dept as (
select
dname as dname,
json_agg(
jsonb_build_object(
'ename', ename,
'is_happy', dname = 'management'
) order by ename
) as emps,
count(*) as empcount
from emps
group by dname
)
select d.*, eg.emps, eg.empcount
from depts d
join emps_grouped_by_dept eg on d.dname = eg.dname
order by dname
fields:
- dname
- created dateTime
- empcount int
- emps *:
table:
fields:
- ename
- is_happy
logbackXml: |
<configuration scan="true" scanPeriod="15 seconds" >
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d %highlight(%-5level) %-15logger{15} -%kvp- %msg%n%ex{0}</pattern>
</encoder>
</appender>
<logger name="com.zaxxer.hikari" level="warn"/>
<logger name="dagr.data" level="info"/>
<logger name="dagr.http" level="warn"/>
<logger name="dagr.jdbc" level="warn"/>
<logger name="dagr.jdbc.sql" level="warn"/>
<logger name="org.quartz" level="warn"/>
<root level="info">
<appender-ref ref="STDOUT" />
</root>
</configuration>
---
apiVersion: v1
kind: Secret
metadata:
name: dagr-connector-secrets
namespace: dagr
type: Opaque
data:
PROVIDER_PRIVATE_KEY: cHJvdmlkZXJfcHJpdmF0ZV9rZXk=
PROVIDER_CLIENT_ID: cHJvdmlkZXJfY2xpZW50X2lk
DB_USER: ZGJfdXNlcg==
DB_PASSWORD: ZGJfcGFzc3dvcmQ=
cd ~/dagr
kubectl apply -f example.yaml
namespace/dagr created
deployment.apps/dagr-connector created
configmap/dagr-connector-config-files created
secret/dagr-connector-secrets created
kubectl get pods -n dagr
kubectl get cm -n dagr
kubectl get secret -n dagr
kubectl logs -n dagr deployment/dagr-connector
Žurnālfaila debug režīma ieslēgšana
kubectl get pods -n dagr
kubectl exec -it dagr-connector-<atbilstoši iepriekšējas komandas rezultātam> -n dagr -- bash
nano logback.xml
kubectl delete -f example.yaml