data-collector-docker

For more information about Data Collector, please refer to the Data Collector documentation.

Build

cd data-collector-project
mvn clean install

Build and release docker-dev image

:bulb: To determine next version manually, check latest version at Docker Hub

:warning: Versioning must follow: https://semver.org

cd data-collector-docker

mvn -B clean install -DskipTests && mvn -B dependency:copy-dependencies

docker login -u USERNAME

docker build --no-cache -t descoped/data-collector:MAJOR.MINOR.PATCH -f ./Dockerfile-dev .
docker push descoped/data-collector:MAJOR.MINOR.PATCH
docker tag descoped/data-collector:MAJOR.MINOR.PATCH descoped/data-collector:latest
docker push descoped/data-collector:latest

Pull and run Data Collector

docker pull descoped/data-collector:latest
docker run -it -p 9990:9990 -v $PWD/conf:/conf -v $PWD/certs:/certs -v /tmp/rawdata:/rawdata descoped/data-collector:latest

Build dev

./build-dev.sh

docker run -it -p 9990:9990 data-collector:dev

Build dev and Visual VM Profiling

Visual VM can be connected to io.descoped.dc.server.Server or using data-collector:dev image.

Follow these instructions for setting up profiling.

Run Server from IntelliJ

1. Configure Server (see enum inside)
2. Consult readme in data-collection-consumer-specifications-repo regarding GCS configuration
3. Run Server in IntelliJ using Visual VM plugin
4. make collect-freg-playground

Docker-dev (data-collection-consumer-specifications)

Consult readme in data-collection-consumer-specifications-repo regarding GCS configuration.

1. make build-data-collector-dev-image
2. make start-gcs-dev
3. make tail-gcs-dev
4. Download and open Visual VM (standard version, not Graal download)
5. File --> Add Remote Host ("0.0.0.0")
6. In left pane, right click "0.0.0.0" and Choose "Add JMX Connection"
7. Set port to 9992
8. make collect-freg-playground

Configuration

Google Secret Manager SSL properties

data.collector.sslBundle.provider=google-secret-manager
data.collector.sslBundle.gcp.projectId=SECRET_MANAGER_PROJECT_ID
data.collector.sslBundle.gcp.serviceAccountKeyPath=(optional)
data.collector.sslBundle.type=(pem | p12)
data.collector.sslBundle.name=ssb-prod-certs
data.collector.sslBundle.publicCertificate=secretName
data.collector.sslBundle.privateCertificate=secretName
data.collector.sslBundle.archiveCertificate=secretName
data.collector.sslBundle.passphrase=secretNam

Rawdata Encryption Credentials

rawdata.encryption.provider = (dynamic-secret-configuration | google-secret-manager)
rawdata.encryption.gcp.projectId => google-secret-manager
rawdata.encryption.gcp.serviceAccountKeyPath => google-secret-manager
rawdata.encryption.key  = (encryptionKey | secretName)
rawdata.encryption.salt = (encryptionSalt | secretName)

Logstash

Set environment variable LOGBACK_CONFIGURATION_FILE=/opt/dc/log4j2-logstash.xml to enable structured logging using Logstash.

.