Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial Kafka support #13190

Merged
merged 148 commits into from
Jun 13, 2024
Merged

Initial Kafka support #13190

merged 148 commits into from
Jun 13, 2024

Conversation

reyesj2
Copy link
Contributor

@reyesj2 reyesj2 commented Jun 12, 2024

No description provided.

reyesj2 and others added 30 commits November 29, 2023 13:34
@reyesj2
Kafka init
86dc7cc 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Update to merge in 2.4/dev
8cf2968 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
merge 2.4/dev
446f1ff 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Removed duplicate kafka setup
26abe90 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@m0duspwnens
Merge remote-tracking branch 'origin/reyesj2/kafka' into kaffytaffy
e25bc8e
@m0duspwnens
add kafka to manager nodes
780ad9e
@m0duspwnens
make manager download so-kafka container
f7534a0
@reyesj2
Fix kafka keystore script
1b49c85 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Update kafka to use manager docker registry
b032eed 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
CRLF -> LF
643d483 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Fix kafka storage setup
7f5741c 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Fix typos and fix error related to elasticsearch saltstate being call…
82830c8 
…ed from logstash state. Logstash will be removed from kafkanodes in future

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@m0duspwnens
Merge remote-tracking branch 'origin/reyesj2/kafka' into kaffytaffy
c9bf1c8
@m0duspwnens
put alphabetical
4e142e0
@m0duspwnens
add so-manager to kafka.nodes pillar
639bf05
@m0duspwnens
kafka.nodes pillar to lf
8e47cc7
@m0duspwnens
listen on 0.0.0.0 for CONTROLLER
db106f8
@m0duspwnens
add kafka nodes as extra hosts for the container
c3f02a6
@m0duspwnens
allow sensor and searchnode to connect to manager kafka ports
ed6137a
@m0duspwnens
port 9093 for kafka docker
18f95e8
@m0duspwnens
kafka broker and listener on 0.0.0.0
b863060
@reyesj2
Set kafka heap size. To be later configured from SOC
dc27bbb 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Generate certs for use with elasticfleet kafka output policy
13105c4 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@m0duspwnens
allow manager to manager on kafka ports
1b8584d
@m0duspwnens
add so-kafka to manager for firewall
7a6b72e
@m0duspwnens
restart container on server config changes
4c5b42b
@reyesj2
Generate kafka keystore on changes to kafka.key
40b08d7 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Add kafka signing_policy for client/server auth. Add kafka-client cer…
436cbc1 
…t on manager so manager can interact with kafka using its own cert

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Merge remote-tracking branch 'origin/kaffytaffy' into reyesj2/kafka
6202090
@reyesj2
Autogenerate kafka topics when a message it sent to non-existing topic
735cfb4 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
reyesj2 added 26 commits May 30, 2024 15:10
@reyesj2
Remove bad config
dbb99d0 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Revert "revert version for soup test before 2.4.80 pipeline unpaused"
00b5a5c 
This reverts commit 48713a4.
@reyesj2
Update kafka nodes pillar before running highstate with pillarwatch e…
e3ea477 
…ngine. This allows configuring your Kafka controllers before cluster comes up for the first time

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka
75bdc92
@reyesj2
Move soup kafka needfuls to up_to_2.4.80
1a832fa 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Remove so-kafka-clusterid script, created during soup
2e85a28 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
update receiver pillar access
d9c58d9 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
update license
fb1d4fd 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
create kafka.id from kafka {partition}-{offset}-{timestamp} for track…
3b0339a 
…ing event

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka
4581a46
@reyesj2
Remove unused kafka reactor
c472326 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Update Kafka controller(s) via SOC UI
284c1be 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
merge 2.4/dev
824f852 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
kafka.id field should only be present when metadata for kafka exists
08557ae 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Added Kafka reset to SOC UI. Incase of changing an active broker to a…
ca7b89c 
… controller topics may become unavailable. Resolving this would require manual intervention. This option allows running a reset to start from a clean slate to then configure cluster to desired state before reenabling Kafka as global pipeline.

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
remove dash(-) from kafka.id
a81e4c3 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
remove redundant 'kafka_' from annotations & defaults
628893f 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Update Kafka self reset & add initial Kafka wrapper scripts to build out
b7eebad 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Revert a whiptail menu change. Kafka is not an install option
dd40962 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Revert a so-setup change. Kafka is not an installable option
b56d497 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Renamed Kafka pillar
83412b8 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Append '-securityonion' to all Security Onion related Kafka topics. A…
e8106be 
…djust logstash to ingest all topics ending in '-securityonion' to avoid having to manually list topic names

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka
2bc53f9
@reyesj2
Use kafka:password for kafka certs
f372b09 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
telegraft should only include jolokia config when Kafka is set as the…
9c31622 
… global.pipeline

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2
wrong location for global.pipeline check
80b1d51 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
@reyesj2 reyesj2 merged commit 9ac7e05 into 2.4/dev Jun 13, 2024
1 of 2 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Jun 13, 2024
@reyesj2 reyesj2 deleted the reyesj2/kafka branch June 17, 2024 19:56
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@m0duspwnens m0duspwnens m0duspwnens approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@reyesj2 @m0duspwnens